Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OOM (Out of Memory) Issues #19

Open
marcinguy opened this issue Mar 26, 2021 · 12 comments
Open

OOM (Out of Memory) Issues #19

marcinguy opened this issue Mar 26, 2021 · 12 comments

Comments

@marcinguy
Copy link

marcinguy commented Mar 26, 2021

Hi,

experiencing some OOM issues while running in Docker on powerful box with 84 GB Ram.

[431166.569778] Out of memory: Killed process 3400451 (sys) total-vm:1160377588kB, anon-rss:77285872kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:164212kB oom_score_adj:0

Tried to set stack.yml

jobs: 1

Is this setting it to one job? I see all CPU cores are in use ....

Anything else I can do to reduce memory usage, slow down the processing but avoid OOM?

How many threads is it actually starting? As many as cores? Can you reference some code link?

Sorry if this is little bit off topic/general, but there is little info about Haskell and your project, or is it me that is Haskell beginner. Those question can help folks running it.

Thanks,

@marcinguy
Copy link
Author

marcinguy commented Mar 26, 2021

Maybe related:
#20

@marcinguy
Copy link
Author

Wondering if:

https://github.com/PLSysSec/sys/blob/821c4d7cf924e68838c128cbe824be46c9955416/src/Static/CheckerConfigDef.hs


      cfgBlockBound          :: Int,
                             -- ^ How many blocks should we explore
      cfgLoopBound           :: Int,
                             -- ^ How many times through loops?

Impact RAM usage and what is the balance, balanced value here for each Checker? Are defaults good for big projects?

@marcinguy
Copy link
Author

If this helps, there are my LL file sizes and occurences:

https://github.com/marcinguy/public/blob/master/files-occurences-sizes.txt

@marcinguy
Copy link
Author

I get results for files under < 1 MB (ll files) on modern laptop (Core i5, 20 GB Ram), with bigger files I seem to get OOMs

@deian
Copy link
Member

deian commented Mar 26, 2021

In our experience the parsing stage is kind of gross and slow. Yes the block and loop bound matter too. I'm not sure there is a bug here. You're asking good questions, but some of these I think you can take a stab at answering them and we'd be happy to incorporate documentation/findings

@marcinguy
Copy link
Author

Thanks. Would love to. Pity I never did anything in Haskell. Seems like stack exec sys +RTS -N1 -- -c heapoob -e ll -d /folder helps a little bit here, untill OOM happens later on P.S Total Haskell beginner

@marcinguy
Copy link
Author

@deian Can you give a hint how to run in on several cores only? Limited resources? So that it does not OOM that quickly?

@mlfbrown
Copy link
Contributor

Aha! This is related to #4 --- @deian is right that we do the dumbest thing right now. If you bring down the block bound it should help, I'm guessing the loop bound should already be low. I can fix this in about three weeks (sorry! have been meaning to for a while!), otherwise what will help is (1) reducing the number of paths that static flags (2) reducing the block bound.

@marcinguy
Copy link
Author

Thanks @mlfbrown @deian

Reduced the block bound to 1, not sure how to do (1) paths that static flags and where is it defined, but it still OOMs.

The destination machine I work on have many cores/CPUs, trying to cap it to several cores (docker run --cpuset-cpus="0-N"), than I hope to have more RAM per cores used. Not sure if this will help, how the threading works in Haskell and Sys. Will update later.

Does Sys, Haskell starts one thread per core based on amount of cores? Memory should be shared between threads, but I guess with more cores, you will need more memory??? Speed will be faster with more cores, but you need more Memory. So I want to sacrifice speed for completeness (no OOMs). Is this assumption correct?

@marcinguy
Copy link
Author

Capping CPUs/cores in Docker seem to work, but Sys starts so many Threads as cores .... will see how this will work. Will update.

@marcinguy
Copy link
Author

Trying this ... will see if this helps.

import Control.Concurrent(setNumCapabilities)

fork :: Checker a b () -> Checker a b ()
fork act = do
  s0 <- get
  void $ liftIO $ do
    setNumCapabilities 1
    forkIO (curTGroup s0) $ evalChecker act s0

Can make a PR to add this as an option to set number of threads vs getting cores amount.

Not sure how to get it from command line parameter and pass it on.

Here is my PR:

#22

@marcinguy
Copy link
Author

marcinguy commented Apr 1, 2021

Nah ... running it now with Bash to skip files when it OOMs and continue where it left off.

On my side the issue is closed. Feel free to close if you wish.

Thanks for your help. Looking forward to future improvements.

#!/bin/bash
export LOG_LEVEL=debug

find /local/ -depth -name "*.done" -exec sh -c 'mv "$1" "${1%.ll.done}.ll"' _ {} \;

while IFS= read -r -d '' -u 9
do
        stack exec sys -- -c uninit -e ll -d "$REPLY" >> /local/uninit_results.txt
        mv "$REPLY" "$REPLY".done
done 9< <( find /local/ -type f -name "*.ll" -exec printf '%s\0' {} + )

find /local/ -depth -name "*.done" -exec sh -c 'mv "$1" "${1%.ll.done}.ll"' _ {} \;

while IFS= read -r -d '' -u 9
do
        stack exec sys -- -c heapoob -e ll -d "$REPLY" >> /local/heapoob_results.txt
        mv "$REPLY" "$REPLY".done
done 9< <( find /local/ -type f -name "*.ll" -exec printf '%s\0' {} + )

find /local/ -depth -name "*.done" -exec sh -c 'mv "$1" "${1%.ll.done}.ll"' _ {} \;

while IFS= read -r -d '' -u 9
do
        stack exec sys -- -c concroob -e ll -d "$REPLY" >> /local/concroob_results.txt
        mv "$REPLY" "$REPLY".done
done 9< <( find /local/ -type f -name "*.ll" -exec printf '%s\0' {} + )

find /local/ -depth -name "*.done" -exec sh -c 'mv "$1" "${1%.ll.done}.ll"' _ {} \;

while IFS= read -r -d '' -u 9
do
        stack exec sys -- -c userinput -e ll -d "$REPLY" >> /local/userinput_results.txt
        mv "$REPLY" "$REPLY".done
done 9< <( find /local/ -type f -name "*.ll" -exec printf '%s\0' {} + )

find /local/ -depth -name "*.done" -exec sh -c 'mv "$1" "${1%.ll.done}.ll"' _ {} \;

while IFS= read -r -d '' -u 9
do
        stack exec sys -- -c uaf  -e ll -d "$REPLY" >> /local/uaf_results.txt
        mv "$REPLY" "$REPLY".done
done 9< <( find /local/ -type f -name "*.ll" -exec printf '%s\0' {} + )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants