Restricted Workers

Documentation for the restricted-workers library.

About

This library provides an abstract interface for running various kinds of workers under resource limitations. The library provides Worker and WorkersPool abstractions for as well as configurable security and resource restrictions settings.

Also see README.md.

Examples/tutorials

Examples of typical usage of the library can be found in the examples/ directory

Security limitations and restrictions

restricted-workers applies a whole lot of limitations to the worker processes, which can be configured using the following datatype:

data LimitSettings = LimitSettings
    { -- | Maximum time for which the code is allowed to run
      -- (in seconds)
      timeout :: Int
      -- | Process priority for the 'nice' syscall.
      -- -20 is the highest, 20 is the lowest
    , niceness :: Int
      -- | Resource limits for the 'setrlimit' syscall
    , rlimits :: Maybe RLimits
      -- | The directory that the evaluator process will be 'chroot'ed
      -- into. Please note that if chroot is applied, all the pathes
      -- in 'EvalSettings' will be calculated relatively to this
      -- value.
    , chrootPath :: Maybe FilePath
      -- | The UID that will be set after the call to chroot.
    , processUid :: Maybe UserID
      -- | SELinux security context under which the worker
      -- process will be running.
    , secontext :: Maybe SecurityContext
      -- | A filepath to the 'tasks' file for the desired cgroup.
      --
      -- For example, if I have mounted the @cpu@ controller at
      -- @/cgroups/cpu/@ and I want the evaluator to be running in the
      -- cgroup 'idiaworkers' then the 'cgroupPath' would be
      -- @/cgroups/cpu/idiaworkers@
    , cgroupPath :: Maybe FilePath
    } deriving (Eq, Show, Generic)

There is also a Default instance for LimitSettings and RLimits with most of the restrictions turned off:

defaultLimits :: LimitSettings
defaultLimits = LimitSettings
    { timeout    = 3
    , niceness   = 10
    , rlimits    = Nothing
    , chrootPath = Nothing
    , processUid = Nothing
    , secontext  = Nothing 
    , cgroupPath = Nothing
    }

Below I'll briefly describe each limitation/restriction with some details.

Timeout

The timeout field specifies how much time (in seconds) the server waits for the worker.

NB: this restriction is not being controlled automatically. You, as a Worker client, is responsible for applying this limitation using the helper workerTimeout function from 'System.Restricted.Worker.Internal'. The reason for this is that controlling this thing require either a multi-threaded environment (which sometimes we can't afford) or spawning additional processes for each worker (which is unreliable).

Example of using workerTimeout:

import Control.Applicative ((<$>))
import Control.Concurrent.Async (race)

getInt :: Worker IOWorker -> Int

myFunc :: Worker IOWorker -> IO (Maybe Int)
myFunc w = do
    either (const Nothing) Just <$> race (workerTimeout w) (getInt w)   

There is also processTimeout from 'System.Restricted.Limits'

Niceness

Niceness is merely the value passed to the nice() syscall, nothing special.

rlimits

The resource limits are controlled by syscalls to setrlimit. The limits itself are defined in the RLimits datatype:

data RLimits = RLimits
    { coreFileSizeLimit :: ResourceLimits
    , cpuTimeLimit      :: ResourceLimits
    , dataSizeLimit     :: ResourceLimits
    , fileSizeLimit     :: ResourceLimits
    , openFilesLimit    :: ResourceLimits
    , stackSizeLimit    :: ResourceLimits
    , totalMemoryLimit  :: ResourceLimits
    } deriving (Eq, Show, Generic)

ResourceLimits itself is defined in System.Posix.Resource. For more information on resource limits see setrlimit(3).

Chrooted jail

In order to restrict the worker process we run it inside the chroot jail. The easiest way to create a fully working jail is to use debootstrap.

mkdir
sudo debootstrap wheezy /idia/run/workers/worker1
sudo chmod <USER> /idia/run/workers/worker1
cd /idia/run/workers/worker1
sudo mkdir -p ./home/<USER>
sudo chown <USER> ./home/<USER>
cd ./home/<USER>

Additionally, if you plan on compiling Haskell code in your worker, you may need to mount a couple of directories and install gcc inside the jail.

mkdir .ghc && sudo mount --bind ~/.ghc .ghc
mkdir .cabal && sudo mount --bind ~/.cabal .cabal
mkdir ghc && sudo mount --bind ~/ghc ghc # ghc libs
cd ../..
sudo chroot .
apt-get install gcc # inside the chroot

Useful links

• http://wiki.debian.org/chroot
• http://wiki.debian.org/Debootstrap

Process uid

This is the uid the worker process will run under. The socket file will also be created by the user with this uid.

SELinux

SELinux (Security-enhanced Linux) is a Linux kernel module providing mechanisms for enforcing fine-grained mandatory access control, brought to you by the creators of infamous PRISM!

SELinux allows the system administrator to control the security of the system by specifying in (modular) policy files the AVC (access vector cache) rules. The SELinux kernel module sits there and monitors all the syscalls and, if it finds something that is not explicitly allowed in the policy, it blocks it. (Well, actually something a little bit different is going on, but for the sake of simplicity I am leaving it out)

Everything on your system -- files, network sockets, file handles, processes, directories -- is labelled with a SELinux security context, which consists of a role, a user name (not related to the regular system user name) and a domain (also called type in some literature). In the policy file you specific which domains are allowed to perform various actions on other domains. A typical piece of the policy file will look like this:

allow myprocess_t self:udp_socket { create connect };
allow myprocess_t bin_t:file { execute };

The first line states that the process from the domain myprocess_t is allowed to create and connect to the UDP sockets of the same domain. The second line allows a process in that domain to execute files of type bin_t (usually files in /bin/ and /usr/bin).

Note: the secontext field actually contains only the security domain. When the worker process is changing the security context it uses the same user/resource as it originally had.

In our SELinux policy for interactive-diagrams we have several domains:

• idia_web_t - the domain under which the scotty-pastebin run
• idia_web_exec_t - the domain of the scotty-pastebin executable and other files associated with that binary
• idia_service_t - the domain under which the eval-service run
• idia_service_exec_t - the domain of the eval-service executable and other files associated with that binary
• idia_service_sock_t - UNIX socket files used for communication
• idia_db_t, idia_pkg_t, idia_web_common_t - database files, packages, html files, templates and other stuff
• idia_worker_env_t - chroot'ed environment in which the worker operates
• idia_restricted_t - the most restricted domain in which the workers run and evaluate code

Useful links

• Redhat SELinux docs
• Gentoo Wiki SELinux tutorial
• /usr/share/selinux/default/include/support/ on your system

Cgroups

Cgroups is the system that can be used to aid the way Linux schedules CPU time/shares, distributes memory to the processes. It does so by organizing processes into hierarchical groups with configured behaviour.

Installing cgroups on debian is somewhat tricky, because the package is a little bit weird.

sudo apt-get install cgroup-bin libcgroup1
sudo cgconfigparser -l ~/interactive-diagrams/cgconfig.conf

For our purposes we have a cgroup called idiaworkers. We also mount the cpu controller on /cgroups/cpu:

$> ls -l /cgroups/cpu/
total 0
-rw-r--r--. 1 root root 0 Jul 12 16:22 cgroup.clone_children
--w--w--w-. 1 root root 0 Jul 12 16:22 cgroup.event_control
-rw-r--r--. 1 root root 0 Jul 12 16:22 cgroup.procs
-rw-r--r--. 1 root root 0 Jul 12 16:22 cpu.shares
drwxr-xr-x. 2 root root 0 Jul 12 16:22 idiaworkers
-rw-r--r--. 1 root root 0 Jul 12 16:22 notify_on_release
-rw-r--r--. 1 root root 0 Jul 12 16:22 release_agent
-rw-r--r--. 1 root root 0 Jul 12 16:22 tasks
$> ls -l /cgroups/cpu/idiaworkers
total 0
-rw-r--r--. 1 root    root 0 Jul 12 16:22 cgroup.clone_children
--w--w--w-. 1 root    root 0 Jul 12 16:22 cgroup.event_control
-rw-r--r--. 1 root    root 0 Jul 12 16:22 cgroup.procs
-rw-r--r--. 1 root    root 0 Jul 12 16:22 cpu.shares
-rw-r--r--. 1 root    root 0 Jul 12 16:22 notify_on_release
-rw-r--r--. 1 vagrant root 0 Jul 14 06:21 tasks

In order to modify how much CPU time our group gets, we write to the cpu.shares file: sudo echo 100 > /cgroups/cpu/idiaworkers/cpu.shares. If we want to add the task/process to the group we simply append the tasks file: echo $PID >> /cgroups/cpu/idiaworkers/tasks. The workers append themselves to the task file automatically (if the cgroup restrictions are enabled in the LimitSettings).

Useful links

• kernel.org docs
• Wikipedia