Deal with CopySpecs that include kernel file system directories better

[bmr-cymru]

Currently scooping up a whole directory tree from /sys or /proc can trigger lots of ugly errors in -v mode. There's also an associated problem of triggering dmesg warnings when we touch deprecated files (e.g. several ipv6 sysctls).

There's a few ways we could deal with this:

• Check read permissions, skip files with --w--w--w-.and truncate/chmod instead
• Add an additional parameter to suppress all IO errors
• Add some kind of path black/whitelisting

The last is probably a more maintainable way to cope with these problems long-term but in the short term the others are much easier to implement with our current structure.

[jhjaggars]

Can we not use the addForbiddenPath method to avoid these paths?

[bmr-cymru]

Yes - that's what I've been doing for e.g. the deprecated sysctls, see commit 8db01b2 on rhel-6 branch. The problem with this approach is it gets hairy when you have a large number of such paths to manage. For now it may be the best way forward but there are a number of deprecated paths to avoid in proc and a much larger set of paths that we shouldn't attempt to copy in sysfs - sys is ten years old now and we really should up our game in terms of the information we collect from there. Right now we have bigger problems (our recursive copy function isn't safe for trees containing looping symlinks for e.g.) but this is something that I think we need to improve.

[bmr-cymru]

This is now about 90% fixed. We now handle symlinks that point to directories (but we don't copy the target by default - that way lies madness) and block and character device nodes and other special node types are now handled properly. This means it's now safe to do e.g.:

        self.add_copy_spec("/sys")
        self.add_copy_spec("/dev")
        self.add_copy_spec("/proc")

There's still some work required to make this robust; in particular scooping up the whole of /sys and /proc will require the global blacklists (Issue #316) to avoid tripping over various pipes and comms channels that hang on read() IO.

[bmr-cymru]

The fix for Issue #266 means that we handle unreadable files correctly. I'll do some more testing on this today and close the issue out if there's nothing more to do.