Support BPM container type

[ansd]

Identify BPM containers by "bpm-" prefix in cgroup-path.

See also cloudfoundry/bpm-release#109 and cloudfoundry/bpm-release@ea7fef1.

This PR makes it possible to monitor BPM containers on BOSH director and BOSH deployed VMs.

I tested this change by co-deploying https://github.com/ansd/sysdig-boshrelease on the BOSH director.

/var/vcap/packages/sysdig/bin/csysdig -pc container.type=bpm

is then identifying processes belonging to BPM containers.

Also, I validated on the director that selecting the Containers View in csysdig is showing the containers corresponding to the output of bpm list.

[gnosek]

Looks fine, unless the BPM container ids are hex strings. Then you'd need to truncate them to 12 hex digits to match other runtimes (and we're about to push a refactor of the cgroup matching code which will simplify the common case quite a bit)

[gnosek]

Do BPM containers have any metadata except for an id?

[ansd]

By metadata, do you mean these runc annotations?
Looking into some config.json as for example /var/vcap/data/bpm/bundles/blobstore/blobstore/config.json, I can't find annotations. Therefore I guess that BPM containers do not have any metadata. @xoebus please correct me if I'm wrong.

[gnosek]

I mean things like Docker has labels (arbitrary key-value pairs), container names, container image names (both human-readable and a sha256 hash).

Looks like e.g. the job/process names encoded in the container id could work as labels.

[gnosek]

Looking at the linked issue, if the container id is a Base-32 encoded string, maybe decode it into the container name (if it cannot contain characters like spaces etc.) or into a label (if it's a completely arbitrary string)?

Edit: according to https://www.pivotaltracker.com/n/projects/2070399/stories/150819404 the decoded id contains a job name and a process name. I think they would make fine container labels :)

[ansd]

@gnosek many thanks for your review :)

Regarding your last comment, the issue I linked is a bit misleading. Before v1.0.3 of BPM, container IDs were base32 encoded. From v1.0.3 onwards, the encoding in my comment above applies. However, according to @xoebus, this encoding format can change any time.

Regarding to put the job name and process name into labels / annotations: This would be a change on BPM side, right? And sysdig would then read those labels? Where would these labels then be used in sysdig?

Before proceeding with this PR, we first need to ensure that the BPM cgroup-path has a somewhat stable format. (Maybe I was a bit too early.) Currently, there is no guarantee at all that the cgroup-path format and the container ID stays the same (see cloudfoundry/bpm-release#109 (comment)).

[gnosek]

Regarding to put the job name and process name into labels / annotations: This would be a change on BPM side, right?

Not really, you can pull the metadata from any place that makes sense for a particular runtime (we're e.g. calling the Docker API or reading from rkt JSON files etc.)

Where would these labels then be used in sysdig?

Hmm I don't see labels exposed to (c)sysdig userspace actually so right now this would benefit users of the commercial platform. However, it should be feasible to add e.g. filters like container.label.bpm_job_name=foo.

Currently, there is no guarantee at all that the cgroup-path format and the container ID stays the same

Depending on the BPM release lifecycle, old versions may be supported for a long time so for full points, we'd have to parse the old formats anyway (see e.g. the at least two different rkt layouts).

Anyway, since I don't pretend to know anything about BPM, I leave these considerations to you. From the sysdig side, the patch looks fine as is.

[xoebus]

TIL that RunC has annotations! I think we can probably annotate the containers with useful information if that would help?

[ansd]

Regarding the labels:
I like the idea to add labels (job name and process name make perfectly sense). But maybe for this PR, let's keep the labels out. Let's first gather some experience analysing BPM containers with sysdig and then decide which labels we find useful to build filters on in userspace. Then, we can add them at a later point in time.

Depending on the BPM release lifecycle, old versions may be supported for a long time so for full points, we'd have to parse the old formats anyway (see e.g. the at least two different rkt layouts).

True. If a new BPM version has some different cgroup-path format, the worst which can happen is that sysdig cannot identifiy the container anymore. In that case, I would need to adapt the parsing.

That said, I would be very happy to bring this patch in.

[ansd]

@gnosek, is there anything I can do to bring this feature in?

[gnosek]

Sorry for taking my time getting back to you :) We're going through some churn in the container code right now but the things I'd ask you before merging would be:
• rebase the patch on top of latest dev branch (I can help with any issues you run into)
• replace the stdlib regexes with e.g. !id.empty() && strspn(id.c_str(), "a-zA-Z0-9._-") == id.size() (it's a long sad story involving compiler bugs and binary compatibility with old distributions)
• test that it still works for you

[ansd]

@gnosek thanks for your response :)

I rebased and removed the stdlib regex dependency.

I tested that it still works by:

• co-locating https://github.com/ansd/sysdig-boshrelease on a deployment using https://github.com/voelzmo/pinger-app-release on both the latest trusty and xenial stemcells using bpm v1.0.4
• on a deployed VM /var/vcap/packages/sysdig/bin/csysdig -pc container.type=bpm identifies the bpm container of the pinger-app.

[gnosek]

LGTM :)