Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

memory leak under lux + dos/win #10

Closed
cwebber opened this issue Jun 1, 2019 · 9 comments

Comments

Projects
None yet
3 participants
@cwebber
Copy link
Contributor

commented Jun 1, 2019

I found that in my dos/win + lux program, I was hitting strange memory leaks. I paired down the problem, and it seems like at this point it probably isn't my fault. Save the following program as memory-eater-win.rkt or whatever:


(require lux
         (prefix-in raart: raart)
         dos/win
         lux/chaos/gui)

(define ((does-nothing) env)
  (let lp ()
    (win-write)
    (lp)))

(struct wingame
  (win)
  #:methods gen:word
  [(define (word-tick gw)
     (wingame (win-boot (wingame-win gw))))
   (define (word-event gw e)
     (match e
       ["q" #f]
       [_ gw]))
   (define (word-output gw)
     (raart:blank 0 0))
   (define (word-fps gw)
     30.0)
   (define (word-return gw)
     (void))])

(define nothing-threads
  (for/list ([i (in-range 900)])
    (does-nothing)))

(define (eats-memory-win-lux)
  (call-with-chaos
   (raart:make-raart)
   (lambda ()
     (fiat-lux (wingame (win-mbr nothing-threads))))))

(define (do-nothing-just-win)
  (let lp ([win (win-mbr nothing-threads)])
    (lp (win-boot win))))

(module+ main
  (match (current-command-line-arguments)
    [(vector "win-lux")
     (eats-memory-win-lux)]
    [(vector "just-win")
     (do-nothing-just-win)]))

Now open htop or your favorite task manager and run it like:

racket memory-eater-win.rkt win-lux

Watch the memory. It should start growing like crazy!

Here's the weird thing: without lux it's just fine!

racket memory-eater-win.rkt just-win

No issue with that one on my end.

What's going on? Is lux doing something that's resulting in old continutations not being gc'ed or something?

@cwebber

This comment has been minimized.

Copy link
Contributor Author

commented Jun 1, 2019

btw I verified that this happens with other chaoses as well (the gui one, at least)

@jeapostrophe

This comment has been minimized.

Copy link
Owner

commented Jun 1, 2019

I find the problem without lux. Compare the program with LUX? set to #f and 'gui or #t

#lang racket/base
(require racket/match
         lux
         (prefix-in raart: raart)
         dos/win
         lux/chaos/gui)

(define LUX? #f) ;; #f 'gui #t

(define ((does-nothing) env)
  (let lp ()
    (win-write)
    (lp)))

(struct wingame
  (win)
  #:methods gen:word
  [(define (word-tick gw)
     (show-memory)
     (wingame (win-boot (wingame-win gw))))
   (define (word-event gw e)
     (match e
       ["q" #f]
       [_ gw]))
   (define (word-output gw)
     (if (eq? LUX? 'gui)
       (λ (w h dc)
         (void))
       (raart:blank 0 0)))
   (define (word-fps gw)
     30.0)
   (define (word-return gw)
     (void))])

(define nothing-threads
  (for/list ([i (in-range 900)])
    (does-nothing)))

(define init-win
  (apply win-mbr nothing-threads))

(define (eats-memory-win-lux)
  (call-with-chaos
   (if (eq? LUX? 'gui) (make-gui) (raart:make-raart))
   (lambda ()
     (fiat-lux (wingame init-win)))))

(define (show-memory)
  (begin (collect-garbage)
         (eprintf "~a\n" (real->decimal-string (/ (current-memory-use) (expt 2 20))))))

(define (do-nothing-just-win)
  (let lp ([win init-win])
    (show-memory)
    (lp (win-boot win))))

(module+ main
  (if LUX?
    (eats-memory-win-lux)
    (do-nothing-just-win)))
@jeapostrophe

This comment has been minimized.

Copy link
Owner

commented Jun 1, 2019

And with just dos/win:

#lang racket/base
(require dos/win)

(define ((does-nothing) env)
  (let lp ()
    (win-write)
    (lp)))

(define nothing-threads
  (for/list ([i (in-range 900)])
    (does-nothing)))

(define init-win
  (apply win-mbr nothing-threads))

(define (show-memory)
  (begin (collect-garbage)
         (eprintf "~a\n" (real->decimal-string (/ (current-memory-use) (expt 2 20))))))

(define (do-nothing-just-win)
  (let lp ([win init-win])
    (show-memory)
    (lp (win-boot win))))

(module+ main
  (do-nothing-just-win))
@jeapostrophe

This comment has been minimized.

Copy link
Owner

commented Jun 1, 2019

And just dos/os2:

#lang racket/base
(require dos/os2)

(define ((does-nothing) st)
  (let lp ()
    (os2-write #f)
    (lp)))

(define nothing-threads
  (for/list ([i (in-range 900)])
    (does-nothing)))

(define (show-memory)
  (begin (collect-garbage)
         (eprintf "~a\n" (real->decimal-string (/ (current-memory-use) (expt 2 20))))))

(define (do-nothing-just-win)
  (let lp ([ps nothing-threads])
    (show-memory)
    (define-values (_ nps)
      (os2-boot (λ (a b) (or a b)) #f ps #f))
    (lp nps)))

(module+ main
  (do-nothing-just-win))
@jeapostrophe

This comment has been minimized.

Copy link
Owner

commented Jun 1, 2019

And just dos:

#lang racket/base
(require dos)

(define nothing-threads
  (for/list ([i (in-range 900)])
    (λ (init-st)
      (let lp ()
        (dos-syscall (λ (k) k))
        (lp)))))

(module+ main
  (let lp ([ps nothing-threads])
    (collect-garbage)
    (eprintf "~a\n" (real->decimal-string (/ (current-memory-use) (expt 2 20))))
    (lp (dos-boot tcons #f ps null))))
@jeapostrophe

This comment has been minimized.

Copy link
Owner

commented Jun 1, 2019

A huge amount of it goes away if I turn off contracts in dos

@jeapostrophe

This comment has been minimized.

Copy link
Owner

commented Jun 1, 2019

I think this is a Racket problem. @mflatt --- Can you look at this?

If you run this program:

#lang racket/base

;; Version 1, using DOS
#;(require dos)

;; Version 2, going direct
(begin
  (define 0x80 (make-continuation-prompt-tag 'dos))

  (define (run-process-until-syscall p st)
    (call-with-continuation-barrier
     (λ ()
       (call-with-continuation-prompt
        (λ () (p st))
        0x80
        (λ (x) x)))))

  (define (dos-syscall k->syscall)
    ;; First we capture our context back to the OS
    (call-with-current-continuation
     (λ (k)
       ;; Then we abort, give it to the OS, along with a syscall
       ;; specification
       (abort-current-continuation 0x80 (k->syscall k)))
     0x80))

  (define USE-OBSCENE-MEMORY? #f)
  (define (map-reduce f + a l)
    (if USE-OBSCENE-MEMORY?
      (cond
        [(null? l) a]
        [else
         (+ (f (car l)) (map-reduce f + a (cdr l)))])
      (cond
        [(null? l)
         a]
        [(pair? l)
         (+ (map-reduce f + a (car l))
            (map-reduce f + a (cdr l)))]
        [else
         (f l)])))

  (define USE-LOTS-OF-MEMORY? #f)
  (define (dos-boot merge-effects last-state ps empty-effects)
    (if USE-LOTS-OF-MEMORY?
      (map-reduce (λ (p) (run-process-until-syscall p last-state))
                  merge-effects
                  empty-effects
                  ps)
      (map (λ (p) (run-process-until-syscall p last-state)) ps))))

(define nothing-threads
  (for/list ([i (in-range 900)])
    (λ (init-st)
      (let lp ()
        (dos-syscall (λ (k) k))
        (lp)))))

(module+ main
  (let lp ([ps nothing-threads])
    (collect-garbage)
    (eprintf "~a\n" (real->decimal-string (/ (current-memory-use) (expt 2 20))))
    (lp (dos-boot cons #f ps null))))

With USE-LOTS-OF-MEMORY? set to #f, it takes 13.76MB always on my machine.

With USE-LOTS-OF-MEMORY? set to #t and USE-OBSCENE-MEMORY? set to #f, it grows slowly and ends up with around 30 MB after 10 seconds on my machine.

With USE-LOTS-OF-MEMORY? set to #t and USE-OBSCENE-MEMORY? set to #t, it grows very fast and ends up with around 200 MB after 10 seconds on my machine.

The OBSCENE version is the weirdest, because the code is really just map with variables in the way of cons and null. At least the LOTS/not OBSCENE version is structurally different than map.

Given that all of these programs construct the same values and have almost identical continuations, it seems really strange that they have such drastically different memory use.

@mflatt

This comment has been minimized.

Copy link

commented Jun 4, 2019

The problem in Jay's version boils down to dead values on the C stack when the stack is copied to capture a continuation. The reliable solution to this problem is Racket CS. Meanwhile, I'll push a change to traditional Racket that mitigates the problem, at least in this case on my machine.

mflatt added a commit to racket/racket that referenced this issue Jun 4, 2019

avoid retaining unneeded runstacks in continuations
While a continuation is set up to avoid retaining runstacks, partly by
storing a prompt ID instead of a prompt record, prompt records can
remain on the C stack and get captured anyway. Mitigate that problem
by making the runstack link weak in some prompt record.

Racket CS doesn't have this problem, of course.

Relevant to jeapostrophe/lux#10
@jeapostrophe

This comment has been minimized.

Copy link
Owner

commented Jun 4, 2019

You're great, @mflatt

It's awesome that CS just has the fixed anyways!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.