PythonSchedulerInternship

Internship at ENS Lyon, working on the creation of Python userspace schedulers using ghOSt

Content

• BindingTool: You hopefully won't need to use this. It's the tool for generating approximate binding sources.
• PythonModule: These are the bindings generated by the tool above, then edited manually, plus additional functions and classes usefull for Python usage.
• PythonFifo: This is a working example of a Python scheduler made using these bindings.
• Notes: Various notes I wrote during the internship, including instruction for future interns and researchers.
• Scripts: Scripts that help automating the testing process by launching and destroying ghOSt schedulers with scripts.

Please refer to the README.md in each folder for install instructions.

Install instructions

ghOSt kernel

Before anything, you will need to have a working ghOSt kernel installed.

You can install it easily using my installer. Please also run the pybind (or plus) target, as it installs Python-specific requirements.

PyBind11

If you installed the kernel another way, don't forget to install PyBind11. It can be install with Python's Pip, package "pybind11" (pip3 install pybind11)

ghOSt userspace

After rebooting, please clone or fork ghost-userspace from Google

Then, add the PythonModule and PythonFifo folders inside the ghost-userspace folder.

Changes in Bazel

It is then mandatory mandatory to edit Bazel files to be able to use PyBind11 and the parts of this project. See the following sections for changes in Bazel files.

ghost-userspace/WORKSPACE

Please add this to expose pybind to Bazel. Replace 3.8 by your current Python version.

new_local_repository(
    name = "pybind11",
    path = "/usr/local/lib/python3.8/dist-packages/pybind11/include",
    build_file_content = """
package(
    default_visibility = [
        "//visibility:public",
    ],
)

cc_library(
    name = "headers",
    srcs = glob(["**/*.h"]),
)
""",
)

ghost-userspace/BUILD

Please add this to create a Bazel target for the Python bindings module. Replace 3.8 by your current Python version.

cc_binary(
    name = "python_module",
    srcs = ["PythonModule/python_bindings.cpp", "PythonModule/python_interface.cpp", "PythonModule/python_interface.hpp"],
    copts = compiler_flags + ["-Wall", "-shared", "-fPIC", "-I/usr/include/python3.8 -Iexternal/pybind11"],
    linkshared=True,
    linkstatic=False,
    deps = [
        "@pybind11//:headers",
        ":agent",
        "@com_google_absl//absl/debugging:symbolize",
        "@com_google_absl//absl/flags:parse",
        "@com_google_absl//absl/strings:str_format",
        "@com_google_absl//absl/time",
    ],
)

Please add this to create a Bazel target for the FIFO Python scheduler example.

py_binary(
    name = "python_fifo",
    srcs = ["PythonFifo/python_fifo.py"],
    data = [":python_module",
        "@linux//:libbpf",
    ],
)

ghost-userspace/third_party/linux.BUILD

Change the libbpf target to this:

# Compiles the libbpf shared library.
make(
    name = "libbpf",
    # This is the library source. This filegroup includes the Makefile.
    lib_source = ":libbpf_source",
    # The Makefile uses other files in the Linux kernel tree outside of its
    # directory during the build process (e.g.,
    # `tools/scripts/Makefile.include`).
    build_data = [":source"],
    # This is the target passed to `make` (i.e., `make libbpf.so`).
    targets = ["libbpf.so"],
    out_shared_libs = ["libbpf.so", "libbpf.so.0"],
    # This copy should be done automatically by the rules_foreign_cc tool, yet
    # it is not. This may happen because the libbpf library is not at the root
    # of the Linux kernel tree. Perhaps the rules_foreign_cc tool makes an
    # assumption that the library source is at the root of the kernel tree,
    # which causes its copy of libbpf.so to fail since it cannot find the static
    # library at the root of the kernel tree.
    #
    # Note: The values of the environment variables below are written to
    # GNUMake.log, so look at that file to inspect them. You can also look at
    # that log to see which other environment variables exist.
    postfix_script = "cp $EXT_BUILD_ROOT/external/linux/tools/lib/bpf/libbpf.so $INSTALLDIR/lib/libbpf.so; " +
    "cp $EXT_BUILD_ROOT/external/linux/tools/lib/bpf/libbpf.so $INSTALLDIR/lib/libbpf.so.0; " +
    # By making the `libbpf` directory and copying the libbpf header files into
    # it, we can have the #include paths in the project prefixed by `libbpf`. In
    # other words, we can do `#include "libbpf/header.h"` instead of
    # `#include "header.h"`. With the latter, it is more confusing to figure out
    # where the header file is and could cause conflicts if a header file in the
    # project has the same name as a header file in libbpf.
    "mkdir $INSTALLDIR/include/libbpf; " +
    "cp $EXT_BUILD_ROOT/external/linux/tools/lib/bpf/*.h $INSTALLDIR/include/libbpf",
    visibility = ["//visibility:public"],
)

Testing

That's it, you should be good to go.

Don't forget to give a look to the relevant README.md before working with something.

Give the Python FIFO scheduler a test run! (see the relevant README.md)