Eliminate redundant parsing of mountinfo #608

inatatsu · 2016-03-02T09:17:03Z

Fixes #631 (and the corresponding issue moby/moby#20851). The solutions here are:

Skip parsing of mountinfo in FindCgroupMountpoint() and FindCgroupMountpointAndRoot() when a given cgroup subsystem is not enabled. The check is done by testing the subsystem is included in a result of ParseCgroupFile().
Break the parsing loop in FindCgroupMountpointDir() when all the cgroup subsystems are found.
Skip parsing of mountinfo in getSelinuxMountPoint() when the mount point is found, instead of invoking mount.GetMounts().
Postpone parsing mountinfo by invoking rootfsParentMountPrivate() and getParentMount() until pivot_root() actually fails.

LK4D4 · 2016-03-04T16:48:55Z

rootfs parent mount change doesn't look like it preserves behaviour. Pivot root can be not called at all.

inatatsu · 2016-03-10T07:16:19Z

@LK4D4 Thanks! Modified to invoke rootfsParentMountPrivate() at the original timing if config.NoPivotRoot is enabled.

inatatsu · 2016-03-14T07:13:08Z

@LK4D4 Do I need any other fix?

crosbymichael · 2016-03-14T20:24:40Z

libcontainer/selinux/selinux.go

+		txt := scanner.Text()
+		sepIdx := strings.Index(txt, " - ")
+		if sepIdx == -1 {
+			return selinuxfs


is this correct to return here if the line is malformed? I'm not sure it should return.

@crosbymichael Thanks! I referred to cgroups.getCgroupMountsHelper() to parse mountinfo. Is it better to continue the loop here for robustness?

Probably, @LK4D4 @mrunalp what do you think?

I think we should continue and just skip the malformed lines.

@crosbymichael @mrunalp Fixed according to your comments. Thanks!

inatatsu · 2016-03-21T15:29:32Z

@crosbymichael Do I need any other fix?

LK4D4 · 2016-03-21T15:42:35Z

LGTM
ping @rhatdan because it changes selinux parsing

rhatdan · 2016-03-21T16:30:52Z

LGTM, although I am a little worried about the fragility of the /proc/self/mountinfo parsing, but perhaps this was just as bad when we were using "github.com/docker/docker/pkg/mount"

inatatsu · 2016-03-24T02:12:25Z

@LK4D4 @rhatdan Thanks. Is this PR ready to be merged?

rhatdan · 2016-03-24T12:45:37Z

Fine with me.

cyphar · 2016-03-24T13:29:14Z

libcontainer/selinux/selinux.go

+		if len(txt) < sepIdx+12 {
+			continue
+		}
+		if txt[sepIdx+3:sepIdx+12] != "selinuxfs" {


Why not strings.Contains() or even just a comment explaining where these magic numbers come from?

This stuff runs like gazillion times per container stat, so even Splits and Contains in the top of Docker CPU usage :(

In that case we should cache it, If we don't already. This only needs to be checked once, only way to change this state is to reboot the machine.

Yup, it would be cool. Thanks for info @rhatdan

Could probably be an option to runc and if passed we can skip this check. We can also do the same for the systemd cgroups check.

@cyphar @LK4D4 Thank you very much for your helpful suggestions. I think eliminating the magic numbers makes sense in this context to run a large number of containers, since the primary overhead of mount.GetMounts() comes from parsing all lines in mountinfo and creating corresponding objects. Fixed to use strings.Contains().

rhatdan · 2016-03-25T12:37:07Z

Looking at the code, this will only be called once. selinuxfs=unknown to start, First call gets set to "", then if it is in SELInux mode it will get set to the path. All future calls will return "" or SELINUXFSPATH. And never call this code.

inatatsu · 2016-04-01T04:54:40Z

@rhatdan Thank you for your suggestion. When SELinux is enabled, getSelinuxMountPoint() seems to be invoked once by every container, along the call path factory.StartInitialization() -> linuxStandardInit.Init() -> label.Init() -> selinux.SelinuxEnabled().

cyphar · 2016-04-01T04:57:15Z

libcontainer/selinux/selinux.go

+	scanner := bufio.NewScanner(f)
+	for scanner.Scan() {
+		txt := scanner.Text()
+		sepIdx := strings.Index(txt, " - ")


I realise this is exactly what the old code did, but Is this field always guaranteed to be -? Why not just look at the 9th field? Or is this to protect against spaces in the mountpoint names?

@cyphar Thanks. Replicated the original comment.

inatatsu · 2016-04-05T02:10:15Z

@cyphar Does this look good to you?

inatatsu · 2016-04-08T02:21:10Z

@LK4D4 @rhatdan @crosbymichael @mrunalp @cyphar Is this PR ready to be merged? Or, is any other fix required?

crosbymichael · 2016-04-19T23:22:52Z

LGTM

Avoid parsing the whole lines of mountinfo after all mountpoints of the target subsytems are found, or when the target subsystem is not enabled. Signed-off-by: Tatsushi Inagaki <e29253@jp.ibm.com>

Avoid parsing the whole lines of mountinfo after the mountpoint is found. Signed-off-by: Tatsushi Inagaki <e29253@jp.ibm.com>

Postpone parsing mountinfo until pivot_root() actually failed Signed-off-by: Tatsushi Inagaki <e29253@jp.ibm.com>

LK4D4 · 2016-04-24T05:30:35Z

LGTM

Fix several format issues found by pdf and html

GordonTheTurtle added the status/0-triage label Mar 2, 2016

inatatsu mentioned this pull request Mar 2, 2016

Eliminate redundant parsing of mountinfo moby/moby#20854

Merged

inatatsu force-pushed the reduce-parsing-mountinfo branch from d144b41 to a81948d Compare March 3, 2016 02:07

inatatsu mentioned this pull request Mar 4, 2016

Running a container becomes slower when a large number of containers are alive moby/moby#20851

Closed

inatatsu force-pushed the reduce-parsing-mountinfo branch 4 times, most recently from 85e9042 to 7351913 Compare March 10, 2016 06:47

inatatsu force-pushed the reduce-parsing-mountinfo branch 2 times, most recently from b4ee7b6 to d259a17 Compare March 14, 2016 04:20

crosbymichael reviewed Mar 14, 2016
View reviewed changes

inatatsu force-pushed the reduce-parsing-mountinfo branch 5 times, most recently from f88d2c2 to 33cde44 Compare March 21, 2016 15:24

inatatsu force-pushed the reduce-parsing-mountinfo branch 3 times, most recently from 447b004 to fbd388d Compare March 24, 2016 02:10

cyphar reviewed Mar 24, 2016
View reviewed changes

inatatsu force-pushed the reduce-parsing-mountinfo branch from fbd388d to 3914af9 Compare March 25, 2016 02:23

inatatsu force-pushed the reduce-parsing-mountinfo branch 5 times, most recently from 2ea6286 to 962a0eb Compare April 1, 2016 04:34

cyphar reviewed Apr 1, 2016
View reviewed changes

inatatsu force-pushed the reduce-parsing-mountinfo branch 2 times, most recently from 57fa063 to be38788 Compare April 5, 2016 01:55

inatatsu force-pushed the reduce-parsing-mountinfo branch 2 times, most recently from 2452f27 to c373395 Compare April 8, 2016 02:02

inatatsu force-pushed the reduce-parsing-mountinfo branch 6 times, most recently from cf4ed0d to 38b37ce Compare April 18, 2016 05:28

inatatsu force-pushed the reduce-parsing-mountinfo branch from 38b37ce to 2368b87 Compare April 20, 2016 02:36

inatatsu added 3 commits April 22, 2016 09:41

Cgroup: reduce redundant parsing of mountinfo

2a1a6cd

Avoid parsing the whole lines of mountinfo after all mountpoints of the target subsytems are found, or when the target subsystem is not enabled. Signed-off-by: Tatsushi Inagaki <e29253@jp.ibm.com>

Selinux: reduce redundant parsing of mountinfo

78e1a4f

Avoid parsing the whole lines of mountinfo after the mountpoint is found. Signed-off-by: Tatsushi Inagaki <e29253@jp.ibm.com>

Rootfs: reduce redundant parsing of mountinfo

eb0a144

Postpone parsing mountinfo until pivot_root() actually failed Signed-off-by: Tatsushi Inagaki <e29253@jp.ibm.com>

inatatsu force-pushed the reduce-parsing-mountinfo branch from 2368b87 to eb0a144 Compare April 22, 2016 00:42

LK4D4 merged commit ae0fc15 into opencontainers:master Apr 24, 2016

rhvgoyal mentioned this pull request Oct 25, 2016

Make parent mount private before bind mounting rootfs #1148

Merged

vishh mentioned this pull request Nov 3, 2016

Docker: Error response from daemon: Container command not found or does not exist kubernetes/kubernetes#35914

Closed

stefanberger pushed a commit to stefanberger/runc that referenced this pull request Sep 8, 2017

Merge pull request opencontainers#608 from hqhq/fix_format_issues

f815650

Fix several format issues found by pdf and html

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Eliminate redundant parsing of mountinfo #608

Eliminate redundant parsing of mountinfo #608

inatatsu commented Mar 2, 2016

LK4D4 commented Mar 4, 2016

inatatsu commented Mar 10, 2016

inatatsu commented Mar 14, 2016

crosbymichael Mar 14, 2016

inatatsu Mar 15, 2016

crosbymichael Mar 15, 2016

mrunalp Mar 16, 2016

inatatsu Mar 17, 2016

inatatsu commented Mar 21, 2016

LK4D4 commented Mar 21, 2016

rhatdan commented Mar 21, 2016

inatatsu commented Mar 24, 2016

rhatdan commented Mar 24, 2016

cyphar Mar 24, 2016

LK4D4 Mar 24, 2016

rhatdan Mar 24, 2016

LK4D4 Mar 24, 2016

mrunalp Mar 24, 2016

inatatsu Apr 1, 2016

rhatdan commented Mar 25, 2016

inatatsu commented Apr 1, 2016

cyphar Apr 1, 2016

inatatsu Apr 1, 2016

inatatsu commented Apr 5, 2016

inatatsu commented Apr 8, 2016

crosbymichael commented Apr 19, 2016

LK4D4 commented Apr 24, 2016

Eliminate redundant parsing of mountinfo #608

Eliminate redundant parsing of mountinfo #608

Conversation

inatatsu commented Mar 2, 2016

LK4D4 commented Mar 4, 2016

inatatsu commented Mar 10, 2016

inatatsu commented Mar 14, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

inatatsu commented Mar 21, 2016

LK4D4 commented Mar 21, 2016

rhatdan commented Mar 21, 2016

inatatsu commented Mar 24, 2016

rhatdan commented Mar 24, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rhatdan commented Mar 25, 2016

inatatsu commented Apr 1, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

inatatsu commented Apr 5, 2016

inatatsu commented Apr 8, 2016

crosbymichael commented Apr 19, 2016

LK4D4 commented Apr 24, 2016