Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add PSI support (threshold for swapin/swapout) #100

Open
yangfl opened this issue Nov 10, 2018 · 34 comments

Comments

@yangfl
Copy link
Contributor

commented Nov 10, 2018

earlyoom is only aware of the amount of free swap, but I've seen many times that with large swap, system stop responding due to heavy swapin/swapout, while earlyoom did nothing since there's still plenty of free swap. Setting a low free swap minimum is no better than completely disabling swap and letting kernel oom to handle low memory situation.

Ref #34

@hakavlad

This comment has been minimized.

Copy link
Contributor

commented Nov 10, 2018

while earlyoom did nothing

What was he supposed to do? Please describe in detail the algorithm you want to implement.

@hakavlad

This comment has been minimized.

Copy link
Contributor

commented Nov 10, 2018

The rate of filling of a swap partition should be a trigger?

@rfjakob

This comment has been minimized.

Copy link
Owner

commented Nov 10, 2018

Seems like the psi (pressure stall information) was merged recently: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=eb414681d5a07d28d2ff90dc05f69ec6b232ebd2
It should bei in linux 4.20. This would probably the best source of information when to start killing things, but it will not be available in older kernels.

@hakavlad

This comment has been minimized.

Copy link
Contributor

commented Nov 18, 2018

Do you plan to support PSI in earlyoom 1.3+?

@rfjakob

This comment has been minimized.

Copy link
Owner

commented Nov 18, 2018

Once my distribution has it, yes! (Fedora)

@rfjakob rfjakob added the enhancement label Dec 1, 2018

@hakavlad

This comment has been minimized.

Copy link
Contributor

commented Dec 24, 2018

Congratulations!
Linux 4.20 released with PSI and is already in Kernel-vanilla-stable
https://fedoraproject.org/wiki/Kernel_Vanilla_Repositories
http://www.leemhuis.info/files/kernel-vanilla/repostatus.txt

@rfjakob

This comment has been minimized.

Copy link
Owner

commented Dec 25, 2018

:)

Reading through https://lwn.net/Articles/759781/ , one problem with psi may be that you have to wait for the thrashing to happen before you can react. Maybe earlyoom will look at both psi and free mem (but much lower threshhold, maybe 1%)

@hakavlad

This comment has been minimized.

Copy link
Contributor

commented Dec 25, 2018

Next q.:
How to find a victim when using psi?

@rfjakob

This comment has been minimized.

Copy link
Owner

commented Dec 25, 2018

I guess same as before, highest oom_score

@hakavlad

This comment has been minimized.

Copy link
Contributor

commented Dec 26, 2018

you have to wait

10 sec maximum

@hakavlad

This comment has been minimized.

Copy link
Contributor

commented Dec 26, 2018

it works

pic

pic

@hakavlad

This comment has been minimized.

Copy link
Contributor

commented Dec 30, 2018

PSI support implemented in nohang. It works pretty fast. The wait is not too long.
https://i.imgur.com/vEJbGMR.png

I just added 2 thresholds:
sigterm_psi = 60
sigkill_psi = 80

It uses some avg10 to quickly respond.

@hakavlad

This comment has been minimized.

Copy link
Contributor

commented Jan 1, 2019

The next problem: PSI mem avg10 slowly falls after corrective action (may need more then 20 sec sleep to prevent multiple killings).

@rfjakob

This comment has been minimized.

Copy link
Owner

commented Jan 1, 2019

Good catch. I guess this means we should use "total" instead of the averaged number.

@hakavlad

This comment has been minimized.

Copy link
Contributor

commented Jan 1, 2019

Seems like total always grows. How we can use it?

@hakavlad

This comment has been minimized.

Copy link
Contributor

commented Jan 1, 2019

@hakavlad

This comment has been minimized.

Copy link
Contributor

commented Jan 1, 2019

Eureka!
avg = (total2 - total1) / time
Now I need test it.

@rfjakob

This comment has been minimized.

Copy link
Owner

commented Jan 1, 2019

Yes exactly, you check if it increases

@hakavlad

This comment has been minimized.

Copy link
Contributor

commented Jan 1, 2019

avg = (total1 - total0) / (time1 - time0) / 10000
It works.

Good news: it can works very quick and reacts if swapping starts.
http://okturing.com/src/5628/body
(in this output avg10 is avg found from formula above)

Bad news: it may give false positives. I just ran FF, and it was killed.
http://okturing.com/src/5630/body

Maybe we should use avg1 or avg2 as middle way.

@hakavlad

This comment has been minimized.

Copy link
Contributor

commented Jan 1, 2019

I will try this settings:
$SIGTERM_AVG = 60
$AVG_TIME = 1
I want to give the user to choose any minimum amount of time for which AVG is calculated

@hakavlad

This comment has been minimized.

Copy link
Contributor

commented Jan 2, 2019

http://okturing.com/src/5636/body
avg1 monitor (or monitor for any time, change avg_min_time)

example of output with not very quick memhogging
http://okturing.com/src/5637/body

avg time: 1.0
PSI mem avg: 0.0
some avg10=2.67 avg60=5.62 avg300=9.39 total=229884775 

avg time: 1.0
PSI mem avg: 0.0
some avg10=2.67 avg60=5.62 avg300=9.39 total=229884775 

avg time: 1.0
PSI mem avg: 0.0
some avg10=2.19 avg60=5.43 avg300=9.32 total=229884775 


# START USING SWAP


avg time: 1.0
PSI mem avg: 16.92
some avg10=2.19 avg60=5.43 avg300=9.32 total=230056914 

avg time: 1.1
PSI mem avg: 84.1
some avg10=8.68 avg60=6.50 avg300=9.52 total=230972454 

avg time: 1.0
PSI mem avg: 87.9
some avg10=8.68 avg60=6.50 avg300=9.52 total=231888109 

avg time: 1.1
PSI mem avg: 99.97
some avg10=23.95 avg60=9.33 avg300=10.09 total=232967719 

avg time: 1.1
PSI mem avg: 99.99
some avg10=23.95 avg60=9.33 avg300=10.09 total=234037558 

avg time: 1.0
PSI mem avg: 94.99
some avg10=37.55 avg60=12.26 avg300=10.70 total=235007672 

avg time: 1.0
PSI mem avg: 98.21
some avg10=37.55 avg60=12.26 avg300=10.70 total=236033919 

avg time: 1.1
PSI mem avg: 99.01
some avg10=48.13 avg60=15.00 avg300=11.28 total=237084562 

avg time: 1.1
PSI mem avg: 71.53
some avg10=48.13 avg60=15.00 avg300=11.28 total=237881926 

avg time: 1.1
PSI mem avg: 74.66
some avg10=53.54 avg60=17.06 avg300=11.74 total=238669873 

avg time: 1.0
PSI mem avg: 91.74
some avg10=53.54 avg60=17.06 avg300=11.74 total=239630293 

avg time: 1.0
PSI mem avg: 95.02
some avg10=59.60 avg60=19.35 avg300=12.25 total=240594804 

avg time: 1.0
PSI mem avg: 99.95
some avg10=59.60 avg60=19.35 avg300=12.25 total=241643845 

avg time: 1.0
PSI mem avg: 98.67
some avg10=66.38 avg60=21.89 avg300=12.83 total=242666213 

avg time: 1.0
PSI mem avg: 49.99
some avg10=66.38 avg60=21.89 avg300=12.83 total=243167690 

avg time: 1.0
PSI mem avg: 84.66
some avg10=67.40 avg60=23.53 avg300=13.24 total=244038897 

avg time: 1.0
PSI mem avg: 70.76
some avg10=67.40 avg60=23.53 avg300=13.24 total=244771122 

avg time: 1.0
PSI mem avg: 92.33
some avg10=69.50 avg60=25.35 avg300=13.68 total=245703671 

avg time: 1.1
PSI mem avg: 99.31
some avg10=74.12 avg60=27.62 avg300=14.24 total=246781382 

avg time: 1.1
PSI mem avg: 99.97
some avg10=74.12 avg60=27.62 avg300=14.24 total=247864738 

avg time: 1.1
PSI mem avg: 99.97
some avg10=78.62 avg60=29.96 avg300=14.82 total=248954990 

avg time: 1.0
PSI mem avg: 99.44
some avg10=78.62 avg60=29.96 avg300=14.82 total=249980594 

avg time: 1.1
PSI mem avg: 99.06
some avg10=82.31 avg60=32.22 avg300=15.40 total=251050410 

avg time: 1.1
PSI mem avg: 95.75
some avg10=82.31 avg60=32.22 avg300=15.40 total=252087053 

avg time: 1.0
PSI mem avg: 97.02
some avg10=84.79 avg60=34.30 avg300=15.95 total=253064046 

avg time: 1.0
PSI mem avg: 99.41
some avg10=84.79 avg60=34.30 avg300=15.95 total=254100130 

avg time: 1.1
PSI mem avg: 98.17
some avg10=87.18 avg60=36.39 avg300=16.51 total=255139188 

avg time: 1.1
PSI mem avg: 84.71
some avg10=87.18 avg60=36.39 avg300=16.51 total=256069110 

avg time: 1.1
PSI mem avg: 83.95
some avg10=86.24 avg60=37.88 avg300=16.96 total=256952564 

avg time: 1.1
PSI mem avg: 98.74
some avg10=86.24 avg60=37.88 avg300=16.96 total=258030294 

avg time: 1.1
PSI mem avg: 99.98
some avg10=88.55 avg60=39.88 avg300=17.52 total=259080932 

avg time: 1.1
PSI mem avg: 80.52
some avg10=88.55 avg60=39.88 avg300=17.52 total=259943098 

avg time: 1.1
PSI mem avg: 60.02
some avg10=88.45 avg60=41.45 avg300=18.00 total=260590551 

avg time: 1.0
PSI mem avg: 84.79
some avg10=88.45 avg60=41.45 avg300=18.00 total=261455903 

avg time: 1.0
PSI mem avg: 77.87
some avg10=85.29 avg60=42.42 avg300=18.36 total=262255772 

avg time: 1.0
PSI mem avg: 86.76
some avg10=84.69 avg60=43.72 avg300=18.80 total=263132918 

avg time: 1.0
PSI mem avg: 89.88
some avg10=84.69 avg60=43.72 avg300=18.80 total=264042776 

avg time: 1.0
PSI mem avg: 69.67
some avg10=83.66 avg60=44.87 avg300=19.21 total=264758629 

avg time: 1.0
PSI mem avg: 70.96
some avg10=83.66 avg60=44.87 avg300=19.21 total=265486766 

avg time: 1.0
PSI mem avg: 88.18
some avg10=82.63 avg60=45.95 avg300=19.61 total=266382658 

avg time: 1.0
PSI mem avg: 82.89
some avg10=82.63 avg60=45.95 avg300=19.61 total=267224852 

avg time: 1.0
PSI mem avg: 83.4
some avg10=82.52 avg60=47.13 avg300=20.04 total=268071133 

avg time: 1.0
PSI mem avg: 84.65
some avg10=82.52 avg60=47.13 avg300=20.04 total=268926961 

avg time: 1.0
PSI mem avg: 86.77
some avg10=82.97 avg60=48.37 avg300=20.48 total=269808080 

avg time: 1.0
PSI mem avg: 87.66
some avg10=82.97 avg60=48.37 avg300=20.48 total=270705197 

avg time: 1.0
PSI mem avg: 83.86
some avg10=83.52 avg60=49.60 avg300=20.93 total=271558446 

avg time: 1.0
PSI mem avg: 91.06
some avg10=83.52 avg60=49.60 avg300=20.93 total=272485445 

avg time: 1.0
PSI mem avg: 89.83
some avg10=84.51 avg60=50.89 avg300=21.40 total=273399845 

avg time: 1.0
PSI mem avg: 99.46
some avg10=84.51 avg60=50.89 avg300=21.40 total=274409564 

avg time: 1.0
PSI mem avg: 95.79
some avg10=86.77 avg60=52.40 avg300=21.91 total=275390717 

avg time: 1.0
PSI mem avg: 84.82
some avg10=86.77 avg60=52.40 avg300=21.91 total=276246728 

avg time: 1.0
PSI mem avg: 99.99
some avg10=87.54 avg60=53.66 avg300=22.39 total=277262795 

avg time: 1.0
PSI mem avg: 98.98
some avg10=87.54 avg60=53.66 avg300=22.39 total=278271242 

avg time: 1.0
PSI mem avg: 92.34
some avg10=89.25 avg60=55.08 avg300=22.90 total=279209716 

avg time: 1.0
PSI mem avg: 90.39
some avg10=89.25 avg60=55.08 avg300=22.90 total=280125479 

avg time: 1.0
PSI mem avg: 96.1
some avg10=89.93 avg60=56.32 avg300=23.37 total=281097948 

avg time: 1.0
PSI mem avg: 90.81
some avg10=89.93 avg60=56.32 avg300=23.37 total=282033579 

avg time: 1.0
PSI mem avg: 94.96
some avg10=90.30 avg60=57.49 avg300=23.84 total=283005589 

avg time: 1.0
PSI mem avg: 99.84
some avg10=90.30 avg60=57.49 avg300=23.84 total=284014404 

avg time: 1.0
PSI mem avg: 95.37
some avg10=91.70 avg60=58.81 avg300=24.35 total=284980212 

avg time: 1.0
PSI mem avg: 95.8
some avg10=91.70 avg60=58.81 avg300=24.35 total=285948039 

avg time: 1.0
PSI mem avg: 99.98
some avg10=92.29 avg60=60.00 avg300=24.83 total=286968119 

avg time: 1.0
PSI mem avg: 99.94
some avg10=92.29 avg60=60.00 avg300=24.83 total=287974126 

avg time: 1.0
PSI mem avg: 97.2
some avg10=93.33 avg60=61.24 avg300=25.34 total=288957779 

avg time: 1.0
PSI mem avg: 93.15
some avg10=93.33 avg60=61.24 avg300=25.34 total=289909118 

avg time: 1.0
PSI mem avg: 85.54
some avg10=93.09 avg60=62.25 avg300=25.79 total=290779470 

avg time: 1.0
PSI mem avg: 92.57
some avg10=93.09 avg60=62.25 avg300=25.79 total=291724998 

avg time: 1.0
PSI mem avg: 99.11
some avg10=92.89 avg60=63.22 avg300=26.24 total=292728724 

avg time: 1.0
PSI mem avg: 99.97
some avg10=92.89 avg60=63.22 avg300=26.24 total=293748081 

avg time: 1.0
PSI mem avg: 99.99
some avg10=93.99 avg60=64.39 avg300=26.74 total=294776929 

avg time: 1.0
PSI mem avg: 99.98
some avg10=93.99 avg60=64.39 avg300=26.74 total=295785163 

avg time: 1.0
PSI mem avg: 99.95
some avg10=94.90 avg60=65.52 avg300=27.24 total=296787807 

avg time: 1.0
PSI mem avg: 99.98
some avg10=94.90 avg60=65.52 avg300=27.24 total=297821363 

avg time: 1.0
PSI mem avg: 98.74
some avg10=95.64 avg60=66.62 avg300=27.73 total=298820563 

# OOM!
@hakavlad

This comment has been minimized.

Copy link
Contributor

commented Jan 2, 2019

nohang output with avg3
http://okturing.com/src/5638/body

@hakavlad

This comment has been minimized.

Copy link
Contributor

commented Jan 2, 2019

sigterm_psi = 60
sigkill_psi = 95
avg_min_time = 4
psi_min_sleep_time_after_action = 16

output:

MemAvail: 1589 M, 79.8 % | SwapFree: 732 M, 100.0 %
MemAvail: 1391 M, 69.9 % | SwapFree: 732 M, 100.0 %
some avg10=0.00 avg60=0.00 avg300=0.00 total=0
PSI mem some avg: 0.0, PSI avg time: 4.1
PSI is OK  or  psi_min_sleep_time_after_action did not pass
MemAvail: 1050 M, 52.8 % | SwapFree: 732 M, 100.0 %
MemAvail:  755 M, 37.9 % | SwapFree: 732 M, 100.0 %
MemAvail:  496 M, 24.9 % | SwapFree: 732 M, 100.0 %
MemAvail:  260 M, 13.1 % | SwapFree: 732 M, 100.0 %
MemAvail:   54 M,  2.7 % | SwapFree: 727 M,  99.3 %
MemAvail:    0 M,  0.0 % | SwapFree: 718 M,  98.1 %
MemAvail:    0 M,  0.0 % | SwapFree: 718 M,  98.1 %
MemAvail:    0 M,  0.0 % | SwapFree: 715 M,  97.6 %
MemAvail:    0 M,  0.0 % | SwapFree: 704 M,  96.1 %
MemAvail:    0 M,  0.0 % | SwapFree: 701 M,  95.8 %
MemAvail:    0 M,  0.0 % | SwapFree: 694 M,  94.8 %
MemAvail:    0 M,  0.0 % | SwapFree: 688 M,  93.9 %
MemAvail:    0 M,  0.0 % | SwapFree: 685 M,  93.6 %
MemAvail:    0 M,  0.0 % | SwapFree: 680 M,  92.8 %
MemAvail:    0 M,  0.0 % | SwapFree: 676 M,  92.3 %
MemAvail:    0 M,  0.0 % | SwapFree: 667 M,  91.1 %
some avg10=9.41 avg60=1.70 avg300=0.35 total=1752059
PSI mem some avg: 41.83, PSI avg time: 4.2
PSI is OK  or  psi_min_sleep_time_after_action did not pass
MemAvail:    0 M,  0.0 % | SwapFree: 661 M,  90.2 %
MemAvail:    0 M,  0.0 % | SwapFree: 656 M,  89.6 %
MemAvail:    0 M,  0.0 % | SwapFree: 653 M,  89.2 %
MemAvail:    0 M,  0.0 % | SwapFree: 648 M,  88.5 %
MemAvail:    0 M,  0.0 % | SwapFree: 641 M,  87.6 %
MemAvail:    0 M,  0.0 % | SwapFree: 638 M,  87.1 %
MemAvail:    0 M,  0.0 % | SwapFree: 631 M,  86.1 %
MemAvail:    0 M,  0.0 % | SwapFree: 626 M,  85.5 %
MemAvail:    0 M,  0.0 % | SwapFree: 621 M,  84.8 %
MemAvail:    0 M,  0.0 % | SwapFree: 621 M,  84.8 %
MemAvail:    0 M,  0.0 % | SwapFree: 612 M,  83.6 %
MemAvail:    0 M,  0.0 % | SwapFree: 606 M,  82.7 %
MemAvail:    0 M,  0.0 % | SwapFree: 599 M,  81.8 %
MemAvail:    0 M,  0.0 % | SwapFree: 596 M,  81.4 %
MemAvail:    0 M,  0.0 % | SwapFree: 590 M,  80.5 %
MemAvail:    0 M,  0.0 % | SwapFree: 586 M,  80.1 %
MemAvail:    0 M,  0.0 % | SwapFree: 582 M,  79.5 %
MemAvail:    0 M,  0.0 % | SwapFree: 578 M,  79.0 %
MemAvail:    0 M,  0.0 % | SwapFree: 576 M,  78.7 %
MemAvail:    0 M,  0.0 % | SwapFree: 571 M,  77.9 %
some avg10=32.08 avg60=6.61 avg300=1.41 total=5262760
PSI mem some avg: 84.16, PSI avg time: 4.2
avg (84.16) > sigterm_psi (60)
Found the victim with highest badness:
  Name:    python3
  PID:     1077
  UID:     1000
  Badness: 663, oom_score: 663, oom_score_adj: 0
  VmSize:  1830 MiB
  VmRSS:   1768 MiB (Anon: 1762 MiB, File: 6 MiB, Shmem: 0 MiB)
  VmSwap:    35 MiB
  CmdLine: python3 /usr/bin/oom-trigger 
Implement corrective action:
  Sending SIGTERM to the victim; OK; response time: 16 ms

Duration of work: 45 sec; number of corrective actions:
  - Send SIGTERM to python3: 1
MemAvail: 1779 M, 89.4 % | SwapFree: 597 M,  81.5 %
MemAvail: 1791 M, 90.0 % | SwapFree: 585 M,  80.0 %
MemAvail: 1801 M, 90.5 % | SwapFree: 574 M,  78.4 %
MemAvail: 1811 M, 91.0 % | SwapFree: 564 M,  77.0 %
MemAvail: 1815 M, 91.2 % | SwapFree: 561 M,  76.6 %
MemAvail: 1814 M, 91.2 % | SwapFree: 561 M,  76.6 %
MemAvail: 1813 M, 91.1 % | SwapFree: 561 M,  76.6 %
some avg10=41.53 avg60=10.19 avg300=2.24 total=7650134
PSI mem some avg: 54.31, PSI avg time: 4.4
PSI is OK  or  psi_min_sleep_time_after_action did not pass
MemAvail: 1813 M, 91.1 % | SwapFree: 561 M,  76.6 %
MemAvail: 1811 M, 91.0 % | SwapFree: 562 M,  76.7 %
MemAvail: 1805 M, 90.7 % | SwapFree: 568 M,  77.5 %
MemAvail: 1798 M, 90.4 % | SwapFree: 574 M,  78.4 %
MemAvail: 1796 M, 90.2 % | SwapFree: 576 M,  78.7 %
MemAvail: 1796 M, 90.2 % | SwapFree: 576 M,  78.7 %
MemAvail: 1796 M, 90.2 % | SwapFree: 576 M,  78.7 %
MemAvail: 1796 M, 90.2 % | SwapFree: 576 M,  78.7 %
MemAvail: 1796 M, 90.2 % | SwapFree: 576 M,  78.7 %
some avg10=39.81 avg60=13.12 avg300=3.04 total=9217960
PSI mem some avg: 35.45, PSI avg time: 4.4
PSI is OK  or  psi_min_sleep_time_after_action did not pass
MemAvail: 1796 M, 90.2 % | SwapFree: 576 M,  78.7 %
MemAvail: 1796 M, 90.2 % | SwapFree: 576 M,  78.7 %
@hakavlad

This comment has been minimized.

Copy link
Contributor

commented Jan 2, 2019

I saw false positives even if
avg_min_time = 4
and
psi_min_sleep_time_after_action = 20.
IMHO avg10 is now the safest for default using.
total need testing

@hakavlad

This comment has been minimized.

Copy link
Contributor

commented Jan 2, 2019

PSI mem some avg10:    0.0 | MemAvail:  235 M, 11.8 % | SwapFree: 731 M,  99.9 %
PSI mem some avg10:    0.0 | MemAvail:  165 M,  8.3 % | SwapFree: 731 M,  99.9 %
PSI mem some avg10:    0.0 | MemAvail:  101 M,  5.1 % | SwapFree: 731 M,  99.9 %
PSI mem some avg10:    0.0 | MemAvail:   49 M,  2.5 % | SwapFree: 731 M,  99.9 %
PSI mem some avg10:    0.0 | MemAvail:   17 M,  0.8 % | SwapFree: 731 M,  99.9 %
PSI mem some avg10:    0.0 | MemAvail:    1 M,  0.0 % | SwapFree: 727 M,  99.2 %
PSI mem some avg10:    0.0 | MemAvail:    0 M,  0.0 % | SwapFree: 717 M,  97.9 %
PSI mem some avg10:   1.99 | MemAvail:    0 M,  0.0 % | SwapFree: 713 M,  97.4 %
PSI mem some avg10:   1.99 | MemAvail:    0 M,  0.0 % | SwapFree: 708 M,  96.7 %
PSI mem some avg10:   1.99 | MemAvail:    0 M,  0.0 % | SwapFree: 703 M,  96.0 %
PSI mem some avg10:   1.99 | MemAvail:    7 M,  0.3 % | SwapFree: 695 M,  95.0 %
PSI mem some avg10:   1.99 | MemAvail:   13 M,  0.6 % | SwapFree: 688 M,  94.0 %
PSI mem some avg10:   1.99 | MemAvail:   19 M,  0.9 % | SwapFree: 681 M,  93.0 %
PSI mem some avg10:   1.99 | MemAvail:   27 M,  1.4 % | SwapFree: 673 M,  91.9 %
PSI mem some avg10:   1.99 | MemAvail:   32 M,  1.6 % | SwapFree: 668 M,  91.2 %
PSI mem some avg10:   1.99 | MemAvail:   34 M,  1.7 % | SwapFree: 664 M,  90.7 %
PSI mem some avg10:  10.32 | MemAvail:   39 M,  2.0 % | SwapFree: 659 M,  90.0 %
PSI mem some avg10:  10.32 | MemAvail:   45 M,  2.2 % | SwapFree: 653 M,  89.1 %
PSI mem some avg10:  10.32 | MemAvail:   49 M,  2.4 % | SwapFree: 648 M,  88.5 %
PSI mem some avg10:  10.32 | MemAvail:   55 M,  2.7 % | SwapFree: 641 M,  87.6 %
PSI mem some avg10:  10.32 | MemAvail:   61 M,  3.1 % | SwapFree: 634 M,  86.6 %
PSI mem some avg10:  10.32 | MemAvail:   64 M,  3.2 % | SwapFree: 632 M,  86.3 %
PSI mem some avg10:  10.32 | MemAvail:   51 M,  2.6 % | SwapFree: 630 M,  86.0 %
PSI mem some avg10:  10.32 | MemAvail:    4 M,  0.2 % | SwapFree: 630 M,  86.0 %
PSI mem some avg10:  10.32 | MemAvail:    0 M,  0.0 % | SwapFree: 621 M,  84.8 %
PSI mem some avg10:  20.05 | MemAvail:    0 M,  0.0 % | SwapFree: 612 M,  83.6 %
PSI mem some avg10:  20.05 | MemAvail:    0 M,  0.0 % | SwapFree: 611 M,  83.5 %
PSI mem some avg10:  20.05 | MemAvail:    0 M,  0.0 % | SwapFree: 607 M,  82.9 %
PSI mem some avg10:  20.05 | MemAvail:    0 M,  0.0 % | SwapFree: 600 M,  82.0 %
PSI mem some avg10:  20.05 | MemAvail:    0 M,  0.0 % | SwapFree: 600 M,  82.0 %
PSI mem some avg10:  20.05 | MemAvail:    1 M,  0.0 % | SwapFree: 591 M,  80.7 %
PSI mem some avg10:  20.05 | MemAvail:    0 M,  0.0 % | SwapFree: 583 M,  79.6 %
PSI mem some avg10:  20.05 | MemAvail:    0 M,  0.0 % | SwapFree: 583 M,  79.6 %
PSI mem some avg10:  20.05 | MemAvail:    0 M,  0.0 % | SwapFree: 582 M,  79.5 %
PSI mem some avg10:  20.05 | MemAvail:    0 M,  0.0 % | SwapFree: 578 M,  78.9 %
PSI mem some avg10:  31.09 | MemAvail:    0 M,  0.0 % | SwapFree: 572 M,  78.1 %
PSI mem some avg10:  31.09 | MemAvail:    0 M,  0.0 % | SwapFree: 567 M,  77.5 %
PSI mem some avg10:  31.09 | MemAvail:    0 M,  0.0 % | SwapFree: 563 M,  76.9 %
PSI mem some avg10:  31.09 | MemAvail:    0 M,  0.0 % | SwapFree: 557 M,  76.1 %
PSI mem some avg10:  31.09 | MemAvail:    0 M,  0.0 % | SwapFree: 556 M,  76.0 %
PSI mem some avg10:  31.09 | MemAvail:    0 M,  0.0 % | SwapFree: 555 M,  75.8 %
PSI mem some avg10:  31.09 | MemAvail:    0 M,  0.0 % | SwapFree: 550 M,  75.2 %
PSI mem some avg10:  31.09 | MemAvail:    0 M,  0.0 % | SwapFree: 548 M,  74.9 %
PSI mem some avg10:  31.09 | MemAvail:    0 M,  0.0 % | SwapFree: 541 M,  73.9 %
PSI mem some avg10:  31.09 | MemAvail:    0 M,  0.0 % | SwapFree: 535 M,  73.1 %
PSI mem some avg10:  31.09 | MemAvail:    0 M,  0.0 % | SwapFree: 532 M,  72.6 %
PSI mem some avg10:  40.85 | MemAvail:    0 M,  0.0 % | SwapFree: 527 M,  71.9 %
PSI mem some avg10:  40.85 | MemAvail:    0 M,  0.0 % | SwapFree: 527 M,  71.9 %
PSI mem some avg10:  40.85 | MemAvail:    0 M,  0.0 % | SwapFree: 527 M,  71.9 %
PSI mem some avg10:  40.85 | MemAvail:    0 M,  0.0 % | SwapFree: 518 M,  70.8 %
PSI mem some avg10:  40.85 | MemAvail:    0 M,  0.0 % | SwapFree: 514 M,  70.2 %
PSI mem some avg10:  40.85 | MemAvail:    0 M,  0.0 % | SwapFree: 507 M,  69.3 %
PSI mem some avg10:  40.85 | MemAvail:    0 M,  0.0 % | SwapFree: 506 M,  69.1 %
PSI mem some avg10:  40.85 | MemAvail:    0 M,  0.0 % | SwapFree: 506 M,  69.1 %
PSI mem some avg10:  40.85 | MemAvail:    0 M,  0.0 % | SwapFree: 506 M,  69.1 %
PSI mem some avg10:  40.85 | MemAvail:    0 M,  0.0 % | SwapFree: 501 M,  68.4 %
PSI mem some avg10:  40.85 | MemAvail:    0 M,  0.0 % | SwapFree: 498 M,  68.0 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 490 M,  66.9 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 485 M,  66.2 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 481 M,  65.7 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 472 M,  64.5 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 466 M,  63.6 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 460 M,  62.8 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 456 M,  62.3 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 449 M,  61.4 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 444 M,  60.6 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 438 M,  59.8 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 437 M,  59.6 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 433 M,  59.1 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 424 M,  57.9 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 424 M,  57.9 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 424 M,  57.9 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 424 M,  57.9 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 421 M,  57.5 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 418 M,  57.1 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 412 M,  56.3 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 409 M,  55.8 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 404 M,  55.2 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 399 M,  54.5 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 398 M,  54.3 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 398 M,  54.3 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 393 M,  53.6 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 391 M,  53.3 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 386 M,  52.7 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 384 M,  52.5 %
avg (62.99) > sigterm_psi (60.0)
Found the victim with highest badness:
  Name:    python3
  PID:     1486
  UID:     1000
  Badness: 621, oom_score: 621, oom_score_adj: 0
  VmSize:  1719 MiB
  VmRSS:   1624 MiB (Anon: 1618 MiB, File: 6 MiB, Shmem: 0 MiB)
  VmSwap:    66 MiB
  CmdLine: python3 /usr/bin/oom-trigger 
Implement corrective action:
  Sending SIGTERM to the victim; OK; response time: 70 ms

Duration of work: 1 min 0 sec; number of corrective actions:
  - Send SIGTERM to python3: 1
PSI mem some avg10:  62.99 | MemAvail: 1623 M, 81.5 % | SwapFree: 450 M,  61.4 %
PSI mem some avg10:  62.99 | MemAvail: 1663 M, 83.6 % | SwapFree: 446 M,  60.9 %
PSI mem some avg10:  62.99 | MemAvail: 1667 M, 83.7 % | SwapFree: 442 M,  60.4 %
PSI mem some avg10:  62.99 | MemAvail: 1667 M, 83.7 % | SwapFree: 442 M,  60.4 %
PSI mem some avg10:  68.43 | MemAvail: 1667 M, 83.7 % | SwapFree: 442 M,  60.4 %
PSI mem some avg10:  68.43 | MemAvail: 1666 M, 83.7 % | SwapFree: 442 M,  60.4 %
PSI mem some avg10:  68.43 | MemAvail: 1666 M, 83.7 % | SwapFree: 442 M,  60.4 %
PSI mem some avg10:  68.43 | MemAvail: 1665 M, 83.7 % | SwapFree: 443 M,  60.5 %
PSI mem some avg10:  68.43 | MemAvail: 1665 M, 83.6 % | SwapFree: 443 M,  60.5 %
PSI mem some avg10:  72.15 | MemAvail: 1664 M, 83.6 % | SwapFree: 443 M,  60.5 %
PSI mem some avg10:  72.15 | MemAvail: 1664 M, 83.6 % | SwapFree: 443 M,  60.5 %
PSI mem some avg10:  72.15 | MemAvail: 1663 M, 83.5 % | SwapFree: 443 M,  60.5 %
PSI mem some avg10:  72.15 | MemAvail: 1662 M, 83.5 % | SwapFree: 443 M,  60.5 %
PSI mem some avg10:  72.15 | MemAvail: 1661 M, 83.5 % | SwapFree: 443 M,  60.5 %
PSI mem some avg10:  71.94 | MemAvail: 1660 M, 83.4 % | SwapFree: 443 M,  60.5 %
PSI mem some avg10:  71.94 | MemAvail: 1660 M, 83.4 % | SwapFree: 443 M,  60.5 %
PSI mem some avg10:  71.94 | MemAvail: 1659 M, 83.3 % | SwapFree: 444 M,  60.6 %
PSI mem some avg10:  71.94 | MemAvail: 1658 M, 83.3 % | SwapFree: 444 M,  60.7 %
PSI mem some avg10:  71.94 | MemAvail: 1657 M, 83.2 % | SwapFree: 445 M,  60.8 %
PSI mem some avg10:  74.67 | MemAvail: 1655 M, 83.2 % | SwapFree: 446 M,  60.9 %
PSI mem some avg10:  74.67 | MemAvail: 1654 M, 83.1 % | SwapFree: 447 M,  61.0 %
PSI mem some avg10:  74.67 | MemAvail: 1653 M, 83.0 % | SwapFree: 447 M,  61.1 %
PSI mem some avg10:  74.67 | MemAvail: 1651 M, 82.9 % | SwapFree: 448 M,  61.2 %
PSI mem some avg10:  74.67 | MemAvail: 1649 M, 82.9 % | SwapFree: 450 M,  61.4 %
PSI mem some avg10:  78.35 | MemAvail: 1648 M, 82.8 % | SwapFree: 450 M,  61.5 %
PSI mem some avg10:  78.35 | MemAvail: 1648 M, 82.8 % | SwapFree: 450 M,  61.5 %
PSI mem some avg10:  78.35 | MemAvail: 1647 M, 82.7 % | SwapFree: 450 M,  61.5 %
PSI mem some avg10:  78.35 | MemAvail: 1647 M, 82.8 % | SwapFree: 450 M,  61.5 %
PSI mem some avg10:  79.56 | MemAvail: 1646 M, 82.7 % | SwapFree: 450 M,  61.5 %
PSI mem some avg10:  79.56 | MemAvail: 1646 M, 82.7 % | SwapFree: 451 M,  61.5 %
PSI mem some avg10:  79.56 | MemAvail: 1645 M, 82.7 % | SwapFree: 451 M,  61.6 %
PSI mem some avg10:  79.56 | MemAvail: 1644 M, 82.6 % | SwapFree: 452 M,  61.7 %
PSI mem some avg10:  79.56 | MemAvail: 1643 M, 82.5 % | SwapFree: 452 M,  61.8 %
PSI mem some avg10:   80.0 | MemAvail: 1642 M, 82.5 % | SwapFree: 453 M,  61.9 %
PSI mem some avg10:   80.0 | MemAvail: 1640 M, 82.4 % | SwapFree: 455 M,  62.1 %
PSI mem some avg10:   80.0 | MemAvail: 1640 M, 82.4 % | SwapFree: 455 M,  62.1 %
PSI mem some avg10:   80.0 | MemAvail: 1638 M, 82.3 % | SwapFree: 455 M,  62.1 %
PSI mem some avg10:   80.0 | MemAvail: 1637 M, 82.2 % | SwapFree: 455 M,  62.2 %
PSI mem some avg10:  79.09 | MemAvail: 1636 M, 82.2 % | SwapFree: 455 M,  62.2 %
PSI mem some avg10:  79.09 | MemAvail: 1637 M, 82.2 % | SwapFree: 455 M,  62.2 %
PSI mem some avg10:  79.09 | MemAvail: 1636 M, 82.2 % | SwapFree: 455 M,  62.2 %
PSI mem some avg10:  79.09 | MemAvail: 1635 M, 82.2 % | SwapFree: 455 M,  62.2 %
PSI mem some avg10:  79.09 | MemAvail: 1635 M, 82.1 % | SwapFree: 455 M,  62.2 %
PSI mem some avg10:  78.53 | MemAvail: 1635 M, 82.1 % | SwapFree: 455 M,  62.2 %
PSI mem some avg10:  78.53 | MemAvail: 1634 M, 82.1 % | SwapFree: 455 M,  62.2 %
PSI mem some avg10:  78.53 | MemAvail: 1634 M, 82.1 % | SwapFree: 455 M,  62.2 %
PSI mem some avg10:  78.53 | MemAvail: 1633 M, 82.1 % | SwapFree: 455 M,  62.2 %
PSI mem some avg10:  78.53 | MemAvail: 1633 M, 82.0 % | SwapFree: 455 M,  62.2 %
PSI mem some avg10:  77.71 | MemAvail: 1633 M, 82.0 % | SwapFree: 456 M,  62.2 %
PSI mem some avg10:  77.71 | MemAvail: 1632 M, 82.0 % | SwapFree: 456 M,  62.2 %
PSI mem some avg10:  77.71 | MemAvail: 1632 M, 82.0 % | SwapFree: 456 M,  62.2 %
PSI mem some avg10:  77.71 | MemAvail: 1632 M, 82.0 % | SwapFree: 456 M,  62.2 %
PSI mem some avg10:  78.67 | MemAvail: 1631 M, 82.0 % | SwapFree: 456 M,  62.2 %
PSI mem some avg10:  78.67 | MemAvail: 1631 M, 82.0 % | SwapFree: 456 M,  62.2 %
PSI mem some avg10:  78.67 | MemAvail: 1631 M, 81.9 % | SwapFree: 456 M,  62.2 %
PSI mem some avg10:  78.67 | MemAvail: 1631 M, 81.9 % | SwapFree: 456 M,  62.3 %
PSI mem some avg10:  78.67 | MemAvail: 1630 M, 81.9 % | SwapFree: 456 M,  62.3 %
PSI mem some avg10:  79.63 | MemAvail: 1630 M, 81.9 % | SwapFree: 456 M,  62.3 %
PSI mem some avg10:  79.63 | MemAvail: 1629 M, 81.9 % | SwapFree: 456 M,  62.3 %
PSI mem some avg10:  79.63 | MemAvail: 1629 M, 81.8 % | SwapFree: 456 M,  62.3 %
PSI mem some avg10:  79.63 | MemAvail: 1628 M, 81.8 % | SwapFree: 456 M,  62.3 %
PSI mem some avg10:  79.63 | MemAvail: 1628 M, 81.8 % | SwapFree: 456 M,  62.3 %
PSI mem some avg10:  81.15 | MemAvail: 1628 M, 81.8 % | SwapFree: 457 M,  62.4 %
PSI mem some avg10:  81.15 | MemAvail: 1627 M, 81.8 % | SwapFree: 457 M,  62.4 %
PSI mem some avg10:  81.15 | MemAvail: 1627 M, 81.7 % | SwapFree: 457 M,  62.4 %
PSI mem some avg10:  81.15 | MemAvail: 1626 M, 81.7 % | SwapFree: 457 M,  62.4 %
PSI mem some avg10:  81.15 | MemAvail: 1625 M, 81.7 % | SwapFree: 457 M,  62.4 %
PSI mem some avg10:  83.84 | MemAvail: 1625 M, 81.6 % | SwapFree: 457 M,  62.5 %
PSI mem some avg10:  83.84 | MemAvail: 1624 M, 81.6 % | SwapFree: 458 M,  62.5 %
PSI mem some avg10:  83.84 | MemAvail: 1624 M, 81.6 % | SwapFree: 458 M,  62.5 %
PSI mem some avg10:  83.84 | MemAvail: 1623 M, 81.5 % | SwapFree: 458 M,  62.5 %
PSI mem some avg10:  83.84 | MemAvail: 1622 M, 81.5 % | SwapFree: 458 M,  62.6 %
PSI mem some avg10:  84.77 | MemAvail: 1622 M, 81.5 % | SwapFree: 458 M,  62.6 %
PSI mem some avg10:  84.77 | MemAvail: 1621 M, 81.4 % | SwapFree: 458 M,  62.6 %
PSI mem some avg10:  84.77 | MemAvail: 1620 M, 81.4 % | SwapFree: 458 M,  62.6 %
PSI mem some avg10:  84.77 | MemAvail: 1620 M, 81.4 % | SwapFree: 459 M,  62.6 %
PSI mem some avg10:  86.26 | MemAvail: 1620 M, 81.4 % | SwapFree: 459 M,  62.6 %
PSI mem some avg10:  86.26 | MemAvail: 1619 M, 81.3 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.26 | MemAvail: 1619 M, 81.3 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.26 | MemAvail: 1618 M, 81.3 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.26 | MemAvail: 1618 M, 81.3 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.39 | MemAvail: 1618 M, 81.3 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.39 | MemAvail: 1618 M, 81.3 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.39 | MemAvail: 1617 M, 81.3 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.39 | MemAvail: 1617 M, 81.2 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.39 | MemAvail: 1617 M, 81.2 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.68 | MemAvail: 1616 M, 81.2 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.68 | MemAvail: 1616 M, 81.2 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.68 | MemAvail: 1616 M, 81.2 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.68 | MemAvail: 1615 M, 81.1 % | SwapFree: 460 M,  62.8 %
PSI mem some avg10:  86.68 | MemAvail: 1614 M, 81.1 % | SwapFree: 460 M,  62.8 %
PSI mem some avg10:  83.12 | MemAvail: 1614 M, 81.1 % | SwapFree: 460 M,  62.8 %
PSI mem some avg10:  83.12 | MemAvail: 1613 M, 81.0 % | SwapFree: 460 M,  62.8 %
PSI mem some avg10:  83.12 | MemAvail: 1613 M, 81.0 % | SwapFree: 460 M,  62.8 %
PSI mem some avg10:  83.12 | MemAvail: 1612 M, 81.0 % | SwapFree: 460 M,  62.8 %
PSI mem some avg10:  83.12 | MemAvail: 1611 M, 81.0 % | SwapFree: 460 M,  62.8 %
avg (79.47) > sigterm_psi (60.0)
Found the victim with highest badness:
  Name:    firefox
  PID:     1045
  UID:     1000
  Badness: 68, oom_score: 68, oom_score_adj: 0
  VmSize:  1843 MiB
  VmRSS:    115 MiB (Anon: 34 MiB, File: 22 MiB, Shmem: 58 MiB)
  VmSwap:    70 MiB
  CmdLine: /usr/lib/firefox/firefox 
Implement corrective action:
  Sending SIGTERM to the victim; OK; response time: 17 ms

Duration of work: 1 min 40 sec; number of corrective actions:
  - Send SIGTERM to python3: 1
  - Send SIGTERM to firefox: 1
PSI mem some avg10:  79.47 | MemAvail: 1611 M, 80.9 % | SwapFree: 460 M,  62.8 %
PSI mem some avg10:  79.47 | MemAvail: 1650 M, 82.9 % | SwapFree: 533 M,  72.7 %
PSI mem some avg10:  79.47 | MemAvail: 1649 M, 82.9 % | SwapFree: 533 M,  72.8 %
PSI mem some avg10:  79.47 | MemAvail: 1649 M, 82.8 % | SwapFree: 533 M,  72.8 %
PSI mem some avg10:  76.85 | MemAvail: 1649 M, 82.8 % | SwapFree: 533 M,  72.8 %
PSI mem some avg10:  76.85 | MemAvail: 1648 M, 82.8 % | SwapFree: 533 M,  72.8 %
PSI mem some avg10:  76.85 | MemAvail: 1647 M, 82.8 % | SwapFree: 534 M,  72.9 %
PSI mem some avg10:  76.85 | MemAvail: 1659 M, 83.4 % | SwapFree: 534 M,  72.9 %
PSI mem some avg10:  79.77 | MemAvail: 1659 M, 83.4 % | SwapFree: 534 M,  72.9 %
PSI mem some avg10:  79.77 | MemAvail: 1659 M, 83.4 % | SwapFree: 534 M,  73.0 %
PSI mem some avg10:  79.77 | MemAvail: 1658 M, 83.3 % | SwapFree: 534 M,  73.0 %
PSI mem some avg10:  79.77 | MemAvail: 1656 M, 83.2 % | SwapFree: 534 M,  73.0 %
PSI mem some avg10:  79.77 | MemAvail: 1656 M, 83.2 % | SwapFree: 534 M,  73.0 %
PSI mem some avg10:  75.65 | MemAvail: 1651 M, 82.9 % | SwapFree: 534 M,  73.0 %
PSI mem some avg10:  75.65 | MemAvail: 1650 M, 82.9 % | SwapFree: 534 M,  73.0 %
PSI mem some avg10:  75.65 | MemAvail: 1649 M, 82.9 % | SwapFree: 535 M,  73.0 %
PSI mem some avg10:  75.65 | MemAvail: 1646 M, 82.7 % | SwapFree: 535 M,  73.0 %
PSI mem some avg10:  68.28 | MemAvail: 1646 M, 82.7 % | SwapFree: 535 M,  73.1 %
PSI mem some avg10:  68.28 | MemAvail: 1645 M, 82.6 % | SwapFree: 535 M,  73.1 %
PSI mem some avg10:  68.28 | MemAvail: 1644 M, 82.6 % | SwapFree: 535 M,  73.1 %
PSI mem some avg10:  68.28 | MemAvail: 1642 M, 82.5 % | SwapFree: 536 M,  73.2 %
PSI mem some avg10:  68.28 | MemAvail: 1641 M, 82.4 % | SwapFree: 536 M,  73.2 %
PSI mem some avg10:  66.24 | MemAvail: 1637 M, 82.3 % | SwapFree: 536 M,  73.2 %
PSI mem some avg10:  66.24 | MemAvail: 1637 M, 82.3 % | SwapFree: 536 M,  73.2 %
PSI mem some avg10:  66.24 | MemAvail: 1633 M, 82.1 % | SwapFree: 536 M,  73.2 %
PSI mem some avg10:  66.24 | MemAvail: 1631 M, 82.0 % | SwapFree: 536 M,  73.2 %
PSI mem some avg10:  63.11 | MemAvail: 1630 M, 81.9 % | SwapFree: 536 M,  73.2 %
PSI mem some avg10:  63.11 | MemAvail: 1628 M, 81.8 % | SwapFree: 536 M,  73.2 %
PSI mem some avg10:  63.11 | MemAvail: 1625 M, 81.6 % | SwapFree: 536 M,  73.2 %
PSI mem some avg10:  63.11 | MemAvail: 1625 M, 81.6 % | SwapFree: 536 M,  73.2 %
PSI mem some avg10:  63.11 | MemAvail: 1623 M, 81.5 % | SwapFree: 536 M,  73.3 %
PSI mem some avg10:   62.0 | MemAvail: 1619 M, 81.4 % | SwapFree: 536 M,  73.3 %
PSI mem some avg10:   62.0 | MemAvail: 1615 M, 81.1 % | SwapFree: 536 M,  73.3 %
PSI mem some avg10:   62.0 | MemAvail: 1612 M, 81.0 % | SwapFree: 536 M,  73.3 %
PSI mem some avg10:   62.0 | MemAvail: 1611 M, 80.9 % | SwapFree: 536 M,  73.3 %
PSI mem some avg10:  56.39 | MemAvail: 1610 M, 80.9 % | SwapFree: 536 M,  73.3 %
PSI mem some avg10:  56.39 | MemAvail: 1605 M, 80.7 % | SwapFree: 536 M,  73.3 %
PSI mem some avg10:  56.39 | MemAvail: 1732 M, 87.0 % | SwapFree: 583 M,  79.6 %
PSI mem some avg10:  56.39 | MemAvail: 1732 M, 87.0 % | SwapFree: 583 M,  79.6 %
PSI mem some avg10:  56.39 | MemAvail: 1732 M, 87.0 % | SwapFree: 583 M,  79.6 %
PSI mem some avg10:  47.98 | MemAvail: 1732 M, 87.0 % | SwapFree: 583 M,  79.6 %
PSI mem some avg10:  47.98 | MemAvail: 1732 M, 87.0 % | SwapFree: 583 M,  79.7 %
PSI mem some avg10:  47.98 | MemAvail: 1732 M, 87.0 % | SwapFree: 583 M,  79.7 %
PSI mem some avg10:  47.98 | MemAvail: 1732 M, 87.0 % | SwapFree: 583 M,  79.7 %
PSI mem some avg10:  41.28 | MemAvail: 1732 M, 87.0 % | SwapFree: 583 M,  79.7 %
PSI mem some avg10:  41.28 | MemAvail: 1732 M, 87.0 % | SwapFree: 583 M,  79.7 %
PSI mem some avg10:  41.28 | MemAvail: 1732 M, 87.0 % | SwapFree: 583 M,  79.7 %
PSI mem some avg10:  41.28 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.7 %
PSI mem some avg10:  33.98 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.7 %
PSI mem some avg10:  33.98 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.8 %
PSI mem some avg10:  33.98 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.8 %
PSI mem some avg10:  33.98 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.8 %
PSI mem some avg10:  28.19 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.8 %
PSI mem some avg10:  28.19 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.8 %
PSI mem some avg10:  28.19 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.8 %
PSI mem some avg10:  28.19 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.8 %
PSI mem some avg10:  23.26 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.8 %
PSI mem some avg10:  23.26 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.8 %
PSI mem some avg10:  23.26 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.8 %
PSI mem some avg10:  23.26 | MemAvail: 1731 M, 87.0 % | SwapFree: 586 M,  80.1 %
PSI mem some avg10:  28.29 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  28.29 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  28.29 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  28.29 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  28.29 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  23.16 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  23.16 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  23.16 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  23.16 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  18.96 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  18.96 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  18.96 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  18.96 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  15.71 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  15.71 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %

PSI is very slow to bounce back.
FF is innocent victim in this output (I used psi_min_sleep_time_after_action = 40).

@rfjakob rfjakob changed the title Add threshold for swapin/swapout Add PSI support (threshold for swapin/swapout) May 4, 2019

@hakavlad

This comment has been minimized.

Copy link
Contributor

commented Jun 4, 2019

Info:
Switching off a swapspace (on HDD if it was not empty) can cause a significant increasing PSI value (I've seen some avg10 = 70 when ran swapoff -a).

@rfjakob

This comment has been minimized.

Copy link
Owner

commented Jul 4, 2019

I have tested a few things with PSI, and it seems pretty difficult to do "the right thing".

The difficult example is this: https://github.com/rfjakob/earlyoom/blob/master/contrib/oomstat/loadshift.txt
You have an app that uses 70% of your RAM but is idle, and you have enough swap free. Then another app wants to use 70% of your RAM. PSI goes crazy, but the situation calms down as the idle app is swapped out. The right thing here is not killing anything.

@hakavlad

This comment has been minimized.

Copy link
Contributor

commented Jul 14, 2019

The maximum PSI values depend on the type of swapspace.
99 for slow HDD, 60-70 for zram and SSD.

Not very long-term exceedances are normal.

Only long-term threshold exceedances are a clear pathology and require a reaction.

The optimal settings are very individual.

The right things are:

  • disable PSI monitoring by default
  • respond to only very long (>60 sec) thresholds exceeding.

Offer the following default settings:

psi_checking_enabled = False
psi_metrics = some_avg10
sigterm_psi_threshold = 60
sigkill_psi_threshold = 90
psi_excess_duration = 60 sec
psi_post_action_delay = 60 sec
@hakavlad

This comment has been minimized.

Copy link
Contributor

commented Aug 16, 2019

@rfjakob see
https://www.kernel.org/doc/html/latest/accounting/psi.html#monitoring-for-pressure-thresholds
and
https://www.kernel.org/doc/html/latest/accounting/psi.html#userspace-monitor-usage-example (using poll() since Linux 5.2)

Users can register triggers and use poll() to be woken up when resource pressure exceeds certain thresholds

@rfjakob

This comment has been minimized.

Copy link
Owner

commented Aug 18, 2019

I have read through hakavlad/nohang#25, oh boy, PSI is such a headache. Although in this case it seems that muqss is interacting badly with PSI.

@hakavlad

This comment has been minimized.

Copy link
Contributor

commented Aug 19, 2019

PSI is such a headache

Only on desktops. But don't worry, rfjakob-sensei: PSI should be very useful for servers. Disable PSI by default, add warnings to the documentation, and PSI will no longer be a headache.

@hakavlad

This comment has been minimized.

Copy link
Contributor

commented Sep 1, 2019

@rfjakob Could you add PSI support in earlyoom, please?

@hakavlad

This comment has been minimized.

Copy link
Contributor

commented Sep 1, 2019

muqss is interacting badly with PSI

No.

Do you use also btrfs? See https://bugzilla.kernel.org/show_bug.cgi?id=196729

hakavlad/nohang#25 (comment)

Yes, I do.

Seems like this problem was with swap on btrfs.

@hakavlad

This comment has been minimized.

Copy link
Contributor

commented Sep 1, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.