Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix some things in Nest #39

Merged
merged 2 commits into from
Dec 18, 2023
Merged

Fix some things in Nest #39

merged 2 commits into from
Dec 18, 2023

Conversation

Byte-Lab
Copy link
Contributor

Julia noticed a few things that can be improved in scx_nest. Address them here

Julia pointed out that our current implementation of r_impatient is
incorrect. r_impatient is meant to be a mechanism for more aggressively
growing the primary nest if a task repeatedly isn't able to find a core.
Right now, we trigger r_impatient if we're not able to find an attached
or previous core in the primary nest, but we _should_ be triggering it
only if we're unable to find _any_ core in the primary nest. Fixing the
implementation to do this drastically decreases how aggressively we grow
the primary nest when r_impatient is in effect.

Reported-by: Julia Lawall <julia.lawall@inria.fr>
Signed-off-by: David Vernet <void@manifault.com>
The core sched code calls select_task_rq() in a few places: the task
wakeup path (typical path), the fork() path, and the exec() path. For
nest scheduling, we don't want to select a core from the nest on the
exec() path. If we were previously able to find an idle core, we would
have found it on the fork() path, so we don't gain much by checking on
the exec() path. In fact, it's actually harmful, because we could
incorrectly blow up the primary nest unnecessarily by bumping the same
task between multiple cores for no reason. Let's just opt-out of
select_task_rq calls on the exec() path.

Suggested-by: Julia Lawall <julia.lawall@inria.fr>
Signed-off-by: David Vernet <void@manifault.com>
@Byte-Lab Byte-Lab requested a review from htejun December 18, 2023 19:55
@sirlucjan
Copy link
Collaborator

The only feedback I can give: built, tested, appears to have no problems.

Cloning into 'scx'...
done.
==> Starting prepare()...
Applying patch rustlite.patch...
patching file scheds/rust/meson.build
patching file scheds/rust/scx_rustlite/.gitignore
patching file scheds/rust/scx_rustlite/Cargo.toml
patching symbolic link scheds/rust/scx_rustlite/LICENSE
patching file scheds/rust/scx_rustlite/build.rs
patching file scheds/rust/scx_rustlite/meson.build
patching file scheds/rust/scx_rustlite/rustfmt.toml
patching file scheds/rust/scx_rustlite/src/bpf/intf.h
patching file scheds/rust/scx_rustlite/src/bpf/main.bpf.c
patching file scheds/rust/scx_rustlite/src/bpf_intf.rs
patching file scheds/rust/scx_rustlite/src/bpf_skel.rs
patching file scheds/rust/scx_rustlite/src/main.rs
Applying patch nest.patch...
patching file scheds/c/scx_nest.bpf.c
patching file scheds/c/scx_nest.bpf.c
==> Starting pkgver()...
==> Starting build()...
+ exec meson setup --prefix /usr --libexecdir lib --sbindir bin --buildtype plain --auto-features enabled --wrap-mode nodownload -D b_lto=true -D b_pie=true -D python.bytecompile=1 . build
The Meson build system
Version: 1.3.0
Source dir: /home/lucjan/Pracownia/Repo/kernel/scx-scheds/scx-scheds-git/src/scx
Build dir: /home/lucjan/Pracownia/Repo/kernel/scx-scheds/scx-scheds-git/src/scx/build
Build type: native build
Project name: sched_ext schedulers
Project version: 0.1.2
C compiler for the host machine: cc (gcc 13.2.1 "cc (GCC) 13.2.1 20231216")
C linker for the host machine: cc ld.bfd 2.41.0
Host machine cpu family: x86_64
Host machine cpu: x86_64
Program clang found: YES (/usr/bin/clang)
Program bpftool found: YES (/usr/bin/bpftool)
Program cargo found: YES (/usr/bin/cargo)
Program /home/lucjan/Pracownia/Repo/kernel/scx-scheds/scx-scheds-git/src/scx/meson-scripts/get_clang_ver found: YES (/home/lucjan/Pracownia/Repo/kernel/scx-scheds/scx-scheds-git/src/scx/meson-scripts/get_clang_ver)
Program /home/lucjan/Pracownia/Repo/kernel/scx-scheds/scx-scheds-git/src/scx/meson-scripts/bpftool_build_skel found: YES (/home/lucjan/Pracownia/Repo/kernel/scx-scheds/scx-scheds-git/src/scx/meson-scripts/bpftool_build_skel)
Program /home/lucjan/Pracownia/Repo/kernel/scx-scheds/scx-scheds-git/src/scx/meson-scripts/get_sys_incls found: YES (/home/lucjan/Pracownia/Repo/kernel/scx-scheds/scx-scheds-git/src/scx/meson-scripts/get_sys_incls)
Program /home/lucjan/Pracownia/Repo/kernel/scx-scheds/scx-scheds-git/src/scx/meson-scripts/cargo_fetch found: YES (/home/lucjan/Pracownia/Repo/kernel/scx-scheds/scx-scheds-git/src/scx/meson-scripts/cargo_fetch)
meson.build:33: WARNING: clang >= 17 recommended (/usr/bin/clang ver=16.0.6)
Found pkg-config: YES (/usr/bin/pkg-config) 2.1.0
Run-time dependency libbpf found: YES 1.3.0
Message: cpu=x86_64 bpf_base_cflags=['-g', '-O2', '-Wall', '-Wno-compare-distinct-pointer-types', '-D__TARGET_ARCH_x86', '-mcpu=v3', '-mlittle-endian', '-idirafter /usr/lib/clang/16/include', '-idirafter /usr/local/include', '-idirafter /usr/include']
Build targets in project: 12

sched_ext schedulers 0.1.2

  User defined options
    auto_features     : enabled
    buildtype         : plain
    libexecdir        : lib
    prefix            : /usr
    sbindir           : bin
    wrap_mode         : nodownload
    python.bytecompile: 1
    b_lto             : true
    b_pie             : true

Found ninja-1.11.1 at /usr/bin/ninja
INFO: autodetecting backend as ninja                                                                                                                                                                                                                                           
INFO: calculating backend command to run: /usr/bin/ninja -C /home/lucjan/Pracownia/Repo/kernel/scx-scheds/scx-scheds-git/src/scx/build
ninja: Entering directory `/home/lucjan/Pracownia/Repo/kernel/scx-scheds/scx-scheds-git/src/scx/build'
[32/32] Generating scheds/rust/scx_layered/scx_layered with a custom command (wrapped by meson to set env)
==> Entering fakeroot environment...
==> Starting package()...
ninja: Entering directory `/home/lucjan/Pracownia/Repo/kernel/scx-scheds/scx-scheds-git/src/scx/build'
[4/4] Generating scheds/rust/scx_rustlite/scx_rustlite with a custom command (wrapped by meson to set env)
Installing scheds/c/scx_simple to /home/lucjan/Pracownia/Repo/kernel/scx-scheds/scx-scheds-git/pkg/scx-scheds-git/usr/bin
Installing scheds/c/scx_qmap to /home/lucjan/Pracownia/Repo/kernel/scx-scheds/scx-scheds-git/pkg/scx-scheds-git/usr/bin
Installing scheds/c/scx_central to /home/lucjan/Pracownia/Repo/kernel/scx-scheds/scx-scheds-git/pkg/scx-scheds-git/usr/bin
Installing scheds/c/scx_pair to /home/lucjan/Pracownia/Repo/kernel/scx-scheds/scx-scheds-git/pkg/scx-scheds-git/usr/bin
Installing scheds/c/scx_flatcg to /home/lucjan/Pracownia/Repo/kernel/scx-scheds/scx-scheds-git/pkg/scx-scheds-git/usr/bin
Installing scheds/c/scx_userland to /home/lucjan/Pracownia/Repo/kernel/scx-scheds/scx-scheds-git/pkg/scx-scheds-git/usr/bin
Installing scheds/c/scx_nest to /home/lucjan/Pracownia/Repo/kernel/scx-scheds/scx-scheds-git/pkg/scx-scheds-git/usr/bin
Running custom install script '/home/lucjan/Pracownia/Repo/kernel/scx-scheds/scx-scheds-git/src/scx/meson-scripts/install_rust_user_scheds'
==> Tidying install...
  -> Removing libtool files...
  -> Purging unwanted files...
  -> Removing static library files...
  -> Stripping unneeded symbols from binaries and libraries...
  -> Compressing man and info pages...
==> Checking for packaging issues...
==> WARNING: Package contains reference to $srcdir
usr/bin/scx_simple
usr/bin/scx_qmap
usr/bin/scx_central
usr/bin/scx_pair
usr/bin/scx_flatcg
usr/bin/scx_userland
usr/bin/scx_nest
usr/bin/scx_layered
usr/bin/scx_rustlite
usr/bin/scx_rusty
==> Creating package "scx-scheds-git"...
  -> Generating .PKGINFO file...
  -> Generating .BUILDINFO file...
  -> Generating .MTREE file...
  -> Compressing package...
==> Leaving fakeroot environment.
==> Signing package(s)...
  -> Created signature file scx-scheds-git-0.1.2.r17.g239d5d1-3-x86_64_v3.pkg.tar.zst.sig.
==> Finished making: scx-scheds-git 0.1.2.r17.g239d5d1-3 (pon, 18 gru 2023, 21:51:18)
❯ sudo scx_nest    
[sudo] hasło użytkownika lucjan: 


Wakeup stats
------------
WAKEUP_ATTACHED=0
WAKEUP_PREV_PRIMARY=0
WAKEUP_FULLY_IDLE_PRIMARY=0
WAKEUP_ANY_IDLE_PRIMARY=0
WAKEUP_FULLY_IDLE_RESERVE=0
WAKEUP_ANY_IDLE_RESERVE=0
WAKEUP_IDLE_OTHER=0


Nest stats
----------
TASK_IMPATIENT=0
PROMOTED_TO_PRIMARY=0
PROMOTED_TO_RESERVED=0
DEMOTED_TO_RESERVED=0
RESERVED_AT_CAPACITY=0
SCHEDULED_COMPACTION=0
CANCELLED_COMPACTION=0
EAGERLY_COMPACTED=0
CALLBACK_COMPACTED=0


Consume stats
-------------
CONSUMED=0
NOT_CONSUMED=2



Masks
-----
PRIMARY  ( 0): | ---- |
RESERVED ( 0): | ---- |
OTHER    ( 0): | ---- |
IDLE     ( 0): | ---- |





Wakeup stats
------------
WAKEUP_ATTACHED=1948
WAKEUP_PREV_PRIMARY=326
WAKEUP_FULLY_IDLE_PRIMARY=0
WAKEUP_ANY_IDLE_PRIMARY=757
WAKEUP_FULLY_IDLE_RESERVE=0
WAKEUP_ANY_IDLE_RESERVE=256
WAKEUP_IDLE_OTHER=5


Nest stats
----------
TASK_IMPATIENT=170
PROMOTED_TO_PRIMARY=256
PROMOTED_TO_RESERVED=4
DEMOTED_TO_RESERVED=252
RESERVED_AT_CAPACITY=0
SCHEDULED_COMPACTION=3120
CANCELLED_COMPACTION=3032
EAGERLY_COMPACTED=252
CALLBACK_COMPACTED=0


Consume stats
-------------
CONSUMED=723
NOT_CONSUMED=3391



Masks
-----
PRIMARY  ( 3): | ***- |
RESERVED ( 1): | ---* |
OTHER    ( 0): | ---- |
IDLE     ( 4): | **** |





Wakeup stats
------------
WAKEUP_ATTACHED=3880
WAKEUP_PREV_PRIMARY=652
WAKEUP_FULLY_IDLE_PRIMARY=0
WAKEUP_ANY_IDLE_PRIMARY=1657
WAKEUP_FULLY_IDLE_RESERVE=0
WAKEUP_ANY_IDLE_RESERVE=536
WAKEUP_IDLE_OTHER=6


Nest stats
----------
TASK_IMPATIENT=733
PROMOTED_TO_PRIMARY=536
PROMOTED_TO_RESERVED=4
DEMOTED_TO_RESERVED=532
RESERVED_AT_CAPACITY=0
SCHEDULED_COMPACTION=6397
CANCELLED_COMPACTION=6193
EAGERLY_COMPACTED=532
CALLBACK_COMPACTED=0


Consume stats
-------------
CONSUMED=2126
NOT_CONSUMED=6967



Masks
-----
PRIMARY  ( 4): | **** |
RESERVED ( 0): | ---- |
OTHER    ( 0): | ---- |
IDLE     ( 4): | **** |





Wakeup stats
------------
WAKEUP_ATTACHED=6229
WAKEUP_PREV_PRIMARY=1050
WAKEUP_FULLY_IDLE_PRIMARY=0
WAKEUP_ANY_IDLE_PRIMARY=2854
WAKEUP_FULLY_IDLE_RESERVE=0
WAKEUP_ANY_IDLE_RESERVE=922
WAKEUP_IDLE_OTHER=8


Nest stats
----------
TASK_IMPATIENT=824
PROMOTED_TO_PRIMARY=922
PROMOTED_TO_RESERVED=4
DEMOTED_TO_RESERVED=918
RESERVED_AT_CAPACITY=0
SCHEDULED_COMPACTION=10586
CANCELLED_COMPACTION=10140
EAGERLY_COMPACTED=918
CALLBACK_COMPACTED=0


Consume stats
-------------
CONSUMED=2799
NOT_CONSUMED=11589

@htejun htejun merged commit 3049d60 into main Dec 18, 2023
@htejun htejun deleted the nest_fixes branch January 10, 2024 19:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants