add_copy_spec_limit() is limited

[bmr-cymru]

The add_copy_spec_limit() interface has always been a bit of a hack. Size limits are applied during setup() to the result of applying glob expansion to the copyspec argument. Paths that pass the test are subsequently passed to add_copy_spec() with no limit. Since the glob results can include directories this is obviously broken for any glob that would include a directory in its expansion.

This can definitely be improved:

• Stat files during setup() and get an estimate of the report size
  • can't easily estimate command output size
• Apply limits during collect()
• Separate global, plugin and file limits

It's not really clear how the limit should be applied to paths that include directories; if we're going to allow the limit interface to be used with directories we should define how it's applied. E.g. collect files in the order returned by sorting the result of os.walk() for each item that matches a copyspec.

Having a global limit (and a default plugin limit) would fix the general concerns raised in Issue #150.

[mbaldessari]

yes this is definitely a needed interface. I was finishing up the PCP plugin (https://github.com/mbaldessari/sosreport/blob/pcp-plugin/sos/plugins/pcp.py) and resorted to recursive directory size calculation as pcp log files can have a flexible structure and are potentially big. For my specific PCP use case a very specific add_directory API like add_copy_directory_limited() would be great as it is very well defined in its scope. (Might clash a bit with the existing _spec API family though).