potential memory leak when editing large #lang brag file
#512
Comments
|
just wanted to chime in. Yesterday at one backend, the backend server was using 18GB of memory.... |
|
Thanks for the heads up. I would definitely like to investigate.
I should improve the documentation for
That would be great! I'd like to be able to see what you're seeing.
It does keep a cache of expansions -- because expansion can be so slow, and various things work on fully-expanded syntax: Check-syntax, finding definition sites, and of course evaluating the program when there isn't any compiled The cache is a hash-table. The hash-table key is the source file pathname. The hash-table value includes the original syntax, the expanded syntax, and also an MD5 digest of the file contents. So, a given source file will remain in the cache "forever", but only one version of it. As a result, if you're working with the same set of source files for 10 minutes vs. 10 hours, I wouldn't expect the cache working set to be any bigger (modulo your source edits over time actually making the expanded result any bigger). That's what I expect. And I've never seen an OOM situation. But maybe I'm not using "heavy" enough examples. So again I'd love to try your file if/when you're able to share that. p.s. I could add time expiry. Initially I'd planned to. But later I managed to feel unsure it would actually provide a significant benefit. I wasn't sure what reasonable expiration times would be. Too long, and expiration is N/A. Too short, and the cache becomes N/A (becomes a complication that is effectively a non-cache). Having said that, assuming the memory growth is due to the cache, and not something else, I'd definitely be willing to look at this again! |
Given there was only a single background process I wondered whether this might be how you had implemented it, good to know.
Given your description it seems like the cache shouldn't be growing in the way I'm seeing, and I will keep this in mind as I hunt down what is going on. Will report back when I have a clearer idea/the file that can trigger the issue. |
|
I added some logging to the syntax cache. And doing a lot of editing in a file, prompting many check-syntaxes. So far, I'm not seeing the cache itself grow. Nor I also tentatively added a field to track the LRU for each entry, and a thread to check at interval and purge things above a certain age. It's not horribly complicated, but, I'm not sure I want to commit and merge that. Let's learn more, if we can, about what exactly you're experiencing...? By the way, not that you should need such a work-around, but you can always So, in parallel to maybe helping me debug this, if you ever see a huge memory use, that's how to CTRL+ALT+DEL -- which is always the best tech support advice. 😄 But in all seriousness I would love to get to the bottom of this. |
This is prompted by issue #512. I'm not sure whether to merge this. Does it really help, or is it just a complication? In any case I'd like to get a clearer understanding of the reported issue, first. If this helps with that, great. If not, maybe it's worth doing as an orthogonal change. But, one step at a time.
Welp. That's what I get for not reading the manual and searching in vain for something like |
Also added some comments about why this probably won't help with whatever is going on with issue #512. As well as approaches that would be more likely to help.
This is prompted by issue #512. I'm not sure whether to merge this. Does it really help, or is it just a complication? In any case I'd like to get a clearer understanding of the reported issue, first. If this helps with that, great. If not, maybe it's worth doing as an orthogonal change. But, one step at a time.
Also added some comments about why this probably won't help with whatever is going on with issue #512. As well as approaches that would be more likely to help.
@tgbugs I just wanted to check in and see if it turned out you were ready/willing to share that example file or repo. If so, it would be a great way to jump-start looking at this, again. (I'd explored some ideas before, but wasn't able to really zero in on what is happening.) If not, that's OK, I can try to think of some other angle from which to tackle it. |
|
@greghendershott will definitely share it, I've just been slammed all Jan, I'll try to get it up this week or next weekend. |
|
@tgbugs No worries. Thanks again for reporting the issue, and for offering to share the file when you can. The last couple days, I have tried to dig into this more, anyway. Still working on the assumption (?) this is due to the expanded syntax caching. I have some changes that allow a major GC to evict items from the cache. But I'm still dog-fooding and thinking. I might merge it, even before getting your example file. My hesitation, so far, is due to:
|
|
OK I found a scenario where I can get memory use to grow indefinitely: Opening a few dozen files that use Typed Racket, with I think this may be due to the implementation of #451. Disabling that seems to avoid the problem.
|
|
EDIT: False alarm. To verify my comment, I opened the example file in Racket-mode (ver 20210110.1607) on Emacs 27.1 and DrRacket. In either situation, the CPU usage spiked to 100% when background expansion was running but dropped down to normal after a while |
|
I left a thumbs-up on that, not to mean "I confirmed that", but thank you for a "torture test case". Even though that one didn't pan out, I appreciate it. So I'll leave the thumbs-up. :) |
This is prompted by issue #512. I'm not sure whether to merge this. Does it really help, or is it just a complication? In any case I'd like to get a clearer understanding of the reported issue, first. If this helps with that, great. If not, maybe it's worth doing as an orthogonal change. But, one step at a time.
Also added some comments about why this probably won't help with whatever is going on with issue #512. As well as approaches that would be more likely to help.
Although I'm not yet sure these changes /solve/ #512 -- I'd like to confirm with an example problem file -- I think they mitigate it. Certainly they help with e.g. opening a few dozen file in the typed-racket-more collection with racket-xp-mode enabled. With that, I see current-memory-use do the usual "sawtooth wave" -- rising until a GC, then falling. Although Racket seems to delay releasing memory back to the OS, until sufficient major GCs take place, eventually I do see that happen, too. The changes: 1. Eliminate hash-table for online-check-syntax. The hash-table could accumulate online-check-syntax items for sources other than the source being checked. Instead use with-intercepted-logging, a parameter, and put the results in the cache-entry. Something like this is probably what I should have done in the first place when implementing #451. 2. Change the syntax cache to be able to evict items on major GC. This works by wrapping the cache entries in an ephemeron whose value is a namespace. This is fairly chunky. Nothing is evicted until a major GC. And then perhaps more than necessary is evicted. However I spent a fair amount of time trying and rejecting some other ideas.
Although I'm not yet sure these changes /solve/ #512 -- I'd like to confirm with an example problem file -- I think they mitigate it. Certainly they help with e.g. opening a few dozen file in the typed-racket-more collection with racket-xp-mode enabled. With that, I see current-memory-use do the usual "sawtooth wave" -- rising until a GC, then falling. Although Racket seems to delay releasing memory back to the OS, until sufficient major GCs take place, eventually I do see that happen, too. The changes: 1. Eliminate hash-table for online-check-syntax. The hash-table could accumulate online-check-syntax items for sources other than the source being checked. Instead use with-intercepted-logging, a parameter, and put the results in the cache-entry. Something like this is probably what I should have done in the first place when implementing #451. 2. Change the syntax cache to be able to evict items on major GC. This works by wrapping the cache entries in an ephemeron whose value is a namespace. This is fairly chunky. Nothing is evicted until a major GC. And then perhaps more than necessary is evicted. However I spent a fair amount of time trying and rejecting some other ideas.
|
I've continued working on this, doing more experiments and tests. I'm going to merge 3e63bdc for release. I think it will at least mitigate what both of you are seeing. It might even solve it. |
|
Commit 3e63bdc is live on MELPA >= 20210202.2145 if that's an easier way to try it. If neither of you has a chance to try this, soon, then of course no worries -- I understand and I'm glad to stand down for awhile. |
|
I've been doing a test on the 20210125 (d1cbeb7) release on melpa with the file that was causing the issue before and I haven't had any issues so far. I will test the new release as soon as I can figure out why use-package insists on ignoring my load-path to the git version in favor of the melpa version. |
|
Ok, I've had the originally offending file up and open for a couple of days running against 3e63bdc and have not encountered the leak, so it seems that it might be fixed. I will report back if I run into the issue again. Thanks! |
|
I added a "stress test" to open 500 .rkt files: (module+ slow-test
(require rackunit
racket/file
racket/path)
(for ([_ 2]) (collect-garbage))
(define least (current-memory-use))
(define most least)
(define count 0)
(for* ([roots (in-list '(("racket.rkt" "typed")
("core.rkt" "typed-racket")
("main.rkt" "racket")))]
[path (in-directory
(path-only
(apply collection-file-path roots)))]
#:when (equal? #"rkt" (filename-extension path)))
(set! count (add1 count))
(check-syntax (path->string path) (file->string path))
(define after (current-memory-use))
(printf "~a, ~a, ~v\n" count after (path->string path))
(set! least (min least after))
(set! most (max most after)))
(printf "Least: ~a\n" least)
(printf "Most: ~a\n" most))The memory use looks like:
So, I think the original issue is now fixed. Before closing I'd love to hear back also from @capfredf who saw even more extreme memory use. p.s. I think the expanded syntax for Typed Racket programs is significantly larger. I'm not sure exactly why (if the syntax objects are more complicated, and/or their scope sets, and/or syntax properties hold references to more/bigger things, and/or other things). Although I'm curious about that I think it's N/A for this issue, it's not as if TR is doing anything wrong. |
|
Thank you for very much @greghendershott! I will report back if I see similar issues arise again |
The list is ordered by MRU. It keeps the first `mru-to-keep` items as non-evictable cache-entry structs; the remainder are evictable ephemerons keyed by namespace. The motivation here is to keep, say, a half-dozen recent items in the cache, while still avoiding generally unbounded growth in memory use in issue #512. This performs well using the memory stress test added in the previous commit. Having said that, I'd like to sleep on it and review it with fresh eyeballs, later, before considering merging. Also, it might be good to make `mru-to-keep` user-configurable, although such options are generally a PITA and it would be better if it could be a fixed magic number or somehow automatically "do the right thing".
The list is ordered by MRU. The first `mru-to-keep` items are non-evictable cache-entry structs; the remainder are evictable ephemerons keyed by namespace. The motivation here is to keep, say, a half-dozen recent items in the cache, while still avoiding generally unbounded growth in memory use as in issue #512. Both cache-set! and cache-get traverse the entire list -- though just once -- so they can remove evicted items, as well as promote some to be non-evictable and demote others to be evictable.
The list is ordered from MRU to LRU. The first `mru-to-keep` items are non-evictable cache-entry structs; the remainder are evictable cache-entry structs wrapped in ephemerons keyed by namespace. The motivation here is to keep at least a small number of recent items in the cache, even when they wouldn't be retained in an ephemeron by a strong reference to a namespace. While still avoiding generally unbounded growth in memory use as in issue #512. [Keep in mind that ephemeron-value may return false quite quickly -- it is NOT the case that the value is retained until the next major GC. A naive experiment where ephemeron-value is called immediately after make-ephemeron may return non-false. But inserting, say, (sleep 1) between the two will allow the ephemeron value to become false.] Both cache-set! and cache-get traverse the entire list -- though just once -- so they can: 1. Remove evicted items from the list. 2. Attempt to promote some to be non-evictable (generally impossible as described above, but harmless to try). 3. Definitely demote others to be evictable.
|
Based on your feedback, and my own usage: I believe this is resolved. I'm going to close it. |
|
One final report to say that I have been editing the offending file again for long periods of time and no issues have been encountered, so further evidence that this has been fixed. |
I'm leaving this as a note that I am seeing what appears to be a memory leak when working with a large
#lang bragfile. I have not been able to pin down the issue, but the behavior is essentially that the background racket process's memory usage slowly increases over time and I usually notice after returning from being away from the computer because my memory usage is about to cause the OOM killer to start hunting. It tops out at around 6gigs resident (a bit odd given that the limit is 2048?). I don't have the large brag file somewhere visible at the moment but will link it here once it is.If I had to hazard a guess about what was going on, it would be that
racket-xp-modeis repeatedly expanding the file and that the expansions are large and are somehow not being cleaned up, but I have zero evidence to support that.Racket version is 7.9 bc.
Emacs version is multiple different builds from the
feature/native-compbranch.The text was updated successfully, but these errors were encountered: