Consider scratch disk sizes in the scheduler

[achimnol]

lablup/backend.ai-agent#70 will allow size configuration & enforcements of per-container scratch directories.
The current manager does not take the disk space into account when scheduling, so we need to improve that.

• manager: Add "scratch" intrinsic resource slot type
• agent
  • Add an intrinsic compute resource plugin, "ScratchDevice" and "ScracthPlugin", to ai.backend.agent.intrinsic
  • Report the space availability info of the disk where the scratch root resides to the manager

┆Issue is synchronized with this Asana task by Unito

[achimnol]

Docker containers (Docker 18.09) use by default 320 MiB of read-writable tmpfs spaces.

tmpfs on /dev type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/kcore type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/keys type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/timer_list type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/sched_debug type tmpfs (rw,nosuid,size=65536k,mode=755)

It is unlikely to happen that user programs fill up all of above spaces, but for system stability, we need to consider such scenario.

So we need to subtract the following values when setting the memory limit for container (#52),

• all default tmpfs sizes
• scratch size if the scratch directory uses tmpfs
• shared memory size if allocated (default: 64 MiB)

[achimnol]

We could list above /proc/ mounts from docker inspect result, but still we need to figure out which are writable and what the size limits are:

"MaskedPaths": [
    "/proc/asound",
    "/proc/acpi",
    "/proc/kcore",
    "/proc/keys",
    "/proc/latency_stats",
    "/proc/timer_list",
    "/proc/timer_stats",
    "/proc/sched_debug",
    "/proc/scsi",
    "/sys/firmware"
],
"ReadonlyPaths": [
    "/proc/bus",
    "/proc/fs",
    "/proc/irq",
    "/proc/sys",
    "/proc/sysrq-trigger"
]

[achimnol]

We can list those in-container /proc mounts by reading from /proc/{host-pid-of-container-process}/mounts from the agent.