metrics: fast footprint #766
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Offering this up as a WIP/RFC right now, as it currently shows up a couple of interesting 'things'... On a small machine (my 2-core (HT disabled) NUC, 32Gb RAM), running this test gets us what look to be VM launch timeouts: I believe launching 10 containers at a time, with a 10s sleep between each batch, is just too slow on my low core count machine. We should discuss if this is a real issue for us, and if we need to look at out grpc timeouts and consider bumping them up etc. Secondly, if I run this on a bigger machine (88core Xeon, 384Gb RAM), then the test happily launches all 200 containers with no ti..........docker: Error response from daemon: OCI runtime create failed: Failed to check if grpc server is working: rpc error: code = DeadlineExceeded desc = context deadline exceeded: unknown. Thus, with the defaults as they are we see anomalies running on both small machines and big machines (and I'm sure the KSM issue will be worse on the smaller machines anyway). For the grpc timeouts, I think we should discuss, probably debug, and debate. |
|
/me goes to add Fixes to commit |
jodh-intel
reviewed
Sep 21, 2018
metrics/density/README.md
Outdated
| # Kata Containers density metrics tests | ||
|
|
||
| This directory contains a number of tests to help measure container | ||
| density. Some measures are based around the PSS of the runtime components, |
There was a problem hiding this comment.
I'd consider defining the term container density. Something like:
to help measure the number of containers that can be run simultaneously, known as container density.
Also how about a definition of PSS?
In fact, it might even be worth creating a doc specifically for abbreviations and acronymns like this that all test docs could refer to?
| @@ -0,0 +1,46 @@ | |||
| # Kata Containers density metrics tests | |||
metrics/density/README.md
Outdated
| and others look at the system level ('free' and '/proc/meminfo' for instance) | ||
| impact. | ||
|
|
||
| ## docker_memory_usage |
There was a problem hiding this comment.
Might be worth putting these test names in the titles in backticks?
metrics/density/README.md
Outdated
|
|
||
| This directory contains a number of tests to help measure container | ||
| density. Some measures are based around the PSS of the runtime components, | ||
| and others look at the system level ('free' and '/proc/meminfo' for instance) |
There was a problem hiding this comment.
I'd use backticks for these two.
metrics/density/README.md
Outdated
| ## docker_memory_usage | ||
|
|
||
| This test measures the PSS footprint of the runtime components whilst | ||
| launching a number of small (busybox) containers. |
There was a problem hiding this comment.
Could link to https://hub.docker.com/_/busybox/?
metrics/density/fast_footprint.sh
Outdated
|
|
||
| # See if KSM is enabled. | ||
| # If so, ammend the test name to reflect that | ||
| # FIXME - needs refactoring - see https://github.com/kata-containers/tests/issues/519 |
There was a problem hiding this comment.
Issue closed so can this comment be dropped?
There was a problem hiding this comment.
heh, that is the one bit of ksm code that did not get pulled out into the lib afaict. Let me do that now then...
metrics/density/fast_footprint.sh
Outdated
| fi | ||
| } | ||
|
|
||
| # helper function to get USS of prcess in arg1 |
There was a problem hiding this comment.
Nit: typo - process. Ditto below.
|
|
||
| echo "Timed out waiting for containers to launch (${t}s)" | ||
| cleanup | ||
| die "Timed out waiting for containers to launch (${t}s)" |
There was a problem hiding this comment.
As we have this die, do we need the echo above with the same message?
There was a problem hiding this comment.
I put the two in as the cleanup can take some time (depending on how many containers one has run). When running interactively I wanted some indication that we'd died/timed out on screen before the final death.
metrics/density/fast_footprint.sh
Outdated
| else | ||
| echo "Waiting for containers to launch (${numcontainers} at ${t}s)" | ||
| fi | ||
| sleep 3 |
There was a problem hiding this comment.
Shouldn't this be sleep $step?
metrics/density/fast_footprint.sh
Outdated
| metrics_json_end_array "Results" | ||
| } | ||
|
|
||
|
|
There was a problem hiding this comment.
Nit: extraneous blank line.
grahamwhaley
force-pushed
the
20180920_fast_footprint
branch
from
e26db5a
to
8f266f3
Compare
|
Ping @grahamwhaley - we gotta whip these wips! 😄 |
grahamwhaley
force-pushed
the
20180920_fast_footprint
branch
from
8f266f3
to
b047d03
Compare
|
Updated. Found a couple of errors in the JSON generation (typos) whilst there. Tested once we got around the grpc timeout issue... |
|
/test |
GabyCT
reviewed
Sep 27, 2018
metrics/density/fast_footprint.sh
Outdated
| # kill_processes_before_start | ||
| } | ||
|
|
||
| # helper function to get USS of prcess in arg1 |
There was a problem hiding this comment.
oops - missed one! thx, fixed, pushed.
|
@grahamwhaley , I ran the script Runtime config filesRuntime default config filesRuntime config file contentsConfig file # Copyright (c) 2017-2018 Intel Corporation
#
# SPDX-License-Identifier: Apache-2.0
#
# XXX: WARNING: this file is auto-generated.
# XXX:
# XXX: Source file: "cli/config/configuration.toml.in"
# XXX: Project:
# XXX: Name: Kata Containers
# XXX: Type: kata
[hypervisor.qemu]
path = "/usr/bin/qemu-lite-system-x86_64"
kernel = "/usr/share/kata-containers/vmlinuz.container"
image = "/usr/share/kata-containers/kata-containers.img"
machine_type = "pc"
# Optional space-separated list of options to pass to the guest kernel.
# For example, use `kernel_params = "vsyscall=emulate"` if you are having
# trouble running pre-2.15 glibc.
#
# WARNING: - any parameter specified here will take priority over the default
# parameter value of the same name used to start the virtual machine.
# Do not set values here unless you understand the impact of doing so as you
# may stop the virtual machine from booting.
# To see the list of default parameters, enable hypervisor debug, create a
# container and look for 'default-kernel-parameters' log entries.
kernel_params = " agent.log=debug"
# Path to the firmware.
# If you want that qemu uses the default firmware leave this option empty
firmware = ""
# Machine accelerators
# comma-separated list of machine accelerators to pass to the hypervisor.
# For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"`
machine_accelerators=""
# Default number of vCPUs per SB/VM:
# unspecified or 0 --> will be set to 1
# < 0 --> will be set to the actual number of physical cores
# > 0 <= number of physical cores --> will be set to the specified number
# > number of physical cores --> will be set to the actual number of physical cores
default_vcpus = 1
# Default maximum number of vCPUs per SB/VM:
# unspecified or == 0 --> will be set to the actual number of physical cores or to the maximum number
# of vCPUs supported by KVM if that number is exceeded
# > 0 <= number of physical cores --> will be set to the specified number
# > number of physical cores --> will be set to the actual number of physical cores or to the maximum number
# of vCPUs supported by KVM if that number is exceeded
# WARNING: Depending of the architecture, the maximum number of vCPUs supported by KVM is used when
# the actual number of physical cores is greater than it.
# WARNING: Be aware that this value impacts the virtual machine's memory footprint and CPU
# the hotplug functionality. For example, `default_maxvcpus = 240` specifies that until 240 vCPUs
# can be added to a SB/VM, but the memory footprint will be big. Another example, with
# `default_maxvcpus = 8` the memory footprint will be small, but 8 will be the maximum number of
# vCPUs supported by the SB/VM. In general, we recommend that you do not edit this variable,
# unless you know what are you doing.
default_maxvcpus = 0
# Bridges can be used to hot plug devices.
# Limitations:
# * Currently only pci bridges are supported
# * Until 30 devices per bridge can be hot plugged.
# * Until 5 PCI bridges can be cold plugged per VM.
# This limitation could be a bug in qemu or in the kernel
# Default number of bridges per SB/VM:
# unspecified or 0 --> will be set to 1
# > 1 <= 5 --> will be set to the specified number
# > 5 --> will be set to 5
default_bridges = 1
# Default memory size in MiB for SB/VM.
# If unspecified then it will be set 2048 MiB.
#default_memory = 2048
#
# Default memory slots per SB/VM.
# If unspecified then it will be set 10.
# This is will determine the times that memory will be hotadded to sandbox/VM.
#memory_slots = 10
# Disable block device from being used for a container's rootfs.
# In case of a storage driver like devicemapper where a container's
# root file system is backed by a block device, the block device is passed
# directly to the hypervisor for performance reasons.
# This flag prevents the block device from being passed to the hypervisor,
# 9pfs is used instead to pass the rootfs.
disable_block_device_use = false
# Block storage driver to be used for the hypervisor in case the container
# rootfs is backed by a block device. This is either virtio-scsi or
# virtio-blk.
block_device_driver = "virtio-scsi"
# Enable iothreads (data-plane) to be used. This causes IO to be
# handled in a separate IO thread. This is currently only implemented
# for SCSI.
#
enable_iothreads = false
# Enable pre allocation of VM RAM, default false
# Enabling this will result in lower container density
# as all of the memory will be allocated and locked
# This is useful when you want to reserve all the memory
# upfront or in the cases where you want memory latencies
# to be very predictable
# Default false
#enable_mem_prealloc = true
# Enable huge pages for VM RAM, default false
# Enabling this will result in the VM memory
# being allocated using huge pages.
# This is useful when you want to use vhost-user network
# stacks within the container. This will automatically
# result in memory pre allocation
#enable_hugepages = true
# Enable swap of vm memory. Default false.
# The behaviour is undefined if mem_prealloc is also set to true
#enable_swap = true
# This option changes the default hypervisor and kernel parameters
# to enable debug output where available. This extra output is added
# to the proxy logs, but only when proxy debug is also enabled.
#
# Default false
enable_debug = true
# Disable the customizations done in the runtime when it detects
# that it is running on top a VMM. This will result in the runtime
# behaving as it would when running on bare metal.
#
#disable_nesting_checks = true
# This is the msize used for 9p shares. It is the number of bytes
# used for 9p packet payload.
#msize_9p = 8192
# If true and vsocks are supported, use vsocks to communicate directly
# with the agent and no proxy is started, otherwise use unix
# sockets and start a proxy to communicate with the agent.
# Default false
#use_vsock = true
# VFIO devices are hotplugged on a bridge by default.
# Enable hotplugging on root bus. This may be required for devices with
# a large PCI bar, as this is a current limitation with hotplugging on
# a bridge. This value is valid for "pc" machine type.
# Default false
#hotplug_vfio_on_root_bus = true
# If host doesn't support vhost_net, set to true. Thus we won't create vhost fds for nics.
# Default false
#disable_vhost_net = true
#
# Default entropy source.
# The path to a host source of entropy (including a real hardware RNG)
# /dev/urandom and /dev/random are two main options.
# Be aware that /dev/random is a blocking source of entropy. If the host
# runs out of entropy, the VMs boot time will increase leading to get startup
# timeouts.
# The source of entropy /dev/urandom is non-blocking and provides a
# generally acceptable source of entropy. It should work well for pretty much
# all practical purposes.
#entropy_source= "/dev/urandom"
[factory]
# VM templating support. Once enabled, new VMs are created from template
# using vm cloning. They will share the same initial kernel, initramfs and
# agent memory by mapping it readonly. It helps speeding up new container
# creation and saves a lot of memory if there are many kata containers running
# on the same host.
#
# When disabled, new VMs are created from scratch.
#
# Default false
#enable_template = true
[proxy.kata]
path = "/usr/libexec/kata-containers/kata-proxy"
# If enabled, proxy messages will be sent to the system log
# (default: disabled)
enable_debug = true
[shim.kata]
path = "/usr/libexec/kata-containers/kata-shim"
# If enabled, shim messages will be sent to the system log
# (default: disabled)
enable_debug = true
[agent.kata]
# There is no field for this section. The goal is only to be able to
# specify which type of agent the user wants to use.
[netmon]
# If enabled, the network monitoring process gets started when the
# sandbox is created. This allows for the detection of some additional
# network being added to the existing network namespace, after the
# sandbox has been created.
# (default: disabled)
#enable_netmon = true
# Specify the path to the netmon binary.
path = "/usr/libexec/kata-containers/kata-netmon"
# If enabled, netmon messages will be sent to the system log
# (default: disabled)
enable_debug = true
[runtime]
# If enabled, the runtime will log additional debug messages to the
# system log
# (default: disabled)
enable_debug = true
#
# Internetworking model
# Determines how the VM should be connected to the
# the container network interface
# Options:
#
# - bridged
# Uses a linux bridge to interconnect the container interface to
# the VM. Works for most cases except macvlan and ipvlan.
#
# - macvtap
# Used when the Container network interface can be bridged using
# macvtap.
internetworking_model="macvtap"
# If enabled, the runtime will create opentracing.io traces and spans.
# (See https://www.jaegertracing.io/docs/getting-started).
# (default: disabled)
#enable_tracing = trueKSM throttlerversionOutput of " systemd serviceImage details---
osbuilder:
url: "https://github.com/kata-containers/osbuilder"
version: "unknown"
rootfs-creation-time: "2018-09-13T22:26:12.472554196+0000Z"
description: "osbuilder rootfs"
file-format-version: "0.0.2"
architecture: "x86_64"
base-distro:
name: "Clear"
version: "25000"
packages:
default:
- "iptables-bin"
- "libudev0-shim"
- "systemd"
extra:
agent:
url: "https://github.com/kata-containers/agent"
name: "kata-agent"
version: "1.3.0-rc1-1ee972176ae437bcace0a37227818c506bb64ba9"
agent-is-init-daemon: "no"Initrd detailsNo initrd LogfilesRuntime logsRecent runtime problems found in system journal: Proxy logsRecent proxy problems found in system journal: Shim logsRecent shim problems found in system journal: Throttler logsNo recent throttler problems found in system journal. Container manager detailsHave DockerOutput of " Output of " Output of " Have KubernetesOutput of " Output of " Output of " Have Output of " PackagesHave Have I ran the |
|
Hi @GabyCT . I suspect the error is eeked out by this test as it is a bit 'harder' than the normal footprint test. This test launches containers in parallel - by default 10 at a time, which can eek out some issues or push some boundaries - for instance, my best guess is that you may be suffering some form of the 'entropy depletion' issue. That was fixed v.recently over at kata-containers/runtime#781 - could you update all your components to the HEAD and see if the issue persists? If it does, we'll have to go debug it. I'm fairly certain it's not a problem with the test itself, but the test may be exposing an issue. |
grahamwhaley
force-pushed
the
20180920_fast_footprint
branch
from
b047d03
to
1e77b92
Compare
|
Oh, hold on - sigh, a rebase went wonky and added an extra commit in there not from the series. sigh. Let me figure out how to fix that! |
The KSM settling detection code was living inside the docker memory test. We want to make use of that detection code in other tests as well, so move it out to the common lib to live with the other KSM related functions. Signed-off-by: Graham Whaley <graham.whaley@intel.com>
grahamwhaley
force-pushed
the
20180920_fast_footprint
branch
from
1e77b92
to
a60728b
Compare
|
Fixed with a rebase - normal service has resumed.... |
|
@grahamwhaley , I'll do a re-test thanks |
|
/test |
|
lgtm
|
|
@intelkevinputnam or @klynnrif could you please check this? thanks |
intelkevinputnam
requested changes
Sep 28, 2018
metrics/density/README.md
Outdated
| containers in parallel and optionally waiting for KSM to settle its memory | ||
| compaction cycles. | ||
|
|
||
| The script is quite configurable via environment variables on which container |
There was a problem hiding this comment.
Try a bulleted list for the options in this paragraph.
There was a problem hiding this comment.
I think I see what you mean @intelkevinputnam - have broken into a list and worded as such. ptal.
grahamwhaley
force-pushed
the
20180920_fast_footprint
branch
from
a60728b
to
8739dbd
Compare
|
@intelkevinputnam ptal |
|
I believe all is ship shape here - looking for another review from @kata-containers/tests |
chavafg
reviewed
Oct 4, 2018
metrics/density/fast_footprint.sh
Outdated
|
|
||
| docker kill $(docker ps -qa) | ||
|
|
||
| # kill_processes_before_start |
There was a problem hiding this comment.
remove this commented line?
metrics/density/fast_footprint.sh
Outdated
| function grab_stats() { | ||
| # If configured, dump the caches so we get a more stable | ||
| # view of what our static footprint really is | ||
| if [[ DUMP_CACHES ]] ; then |
There was a problem hiding this comment.
shouldn't this be: if [[ "$DUMP_CACHES" ]]; then
There was a problem hiding this comment.
oops, yes! That got inherited from the footprint_data.sh, so has been in there for ages. Let me fix them both...
added 5 commits
October 5, 2018 10:40
Add a footprint test that runs faster than the footprint_data test by: - launching containers in parallel - only taking a measure at the start/end, and not at every launch Can also be very useful for: - launching many containers (to test scale out) - testing parallel launches Fixes: kata-containers#767 Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Add a README file giving a brief description and introduction to the available density related tests. Signed-off-by: Graham Whaley <graham.whaley@intel.com>
The ksm enabled check function missed being factored out to the common lib with the other ksm code, and has been breeding. Reduce its population to one copy, in the common library. Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Fix tiny typo in footprint data scripts. Signed-off-by: Graham Whaley <graham.whaley@intel.com>
The check for DUMP_CACHES was incorrect, meaning the cache dump probably always happened. Fix the expression. There was a leftover commented out call and comments in the cleanup routine - remove them. Signed-off-by: Graham Whaley <graham.whaley@intel.com>
grahamwhaley
force-pushed
the
20180920_fast_footprint
branch
from
8739dbd
to
0e0a7d4
Compare
|
Good spots @chavafg - fixed and pushed, in both the fast footprint and the original footprint. |
|
Seems like jenkins started some machines without nested virtualization any idea @mnaser? |
|
lgtm |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add a system level footprint test, that should be faster to run than the
existing footprint_data.sh test. It should be faster as it: