new feature: provide an exclude-list, like rsync

[jti-lanl]

A user of the OS Campagin is using rsync, instead of pfcp, to get a 10x reduction in the amount of data being copied. (They are probably actually losing performance, because pfcp would add ~64x threads running on 3 FTAs, but the point is that they could potentially have the 10x as well as the 64x.)

I imagine we could parse the exclude-list into a read-only vector of regex_t, somewhere in the global options. Perhaps we could compile these once in the master-task initializations, and share out the pre-compiled list as part of options, or else share the uncompiled versions and have every process compile them locally. Either way, we then have a set of per-task compiled exprs.

Some of them might eliminate entire directories, whereas others could match files in multiple directories. So, I think we'd have to run through the list of regexes before worker_readdir() calls opendir(), and then maybe we have to run through the list again, before accepting each result from readdir(), unless we can do something clever to identify those regexes that apply only to directories, and those that apply only to files.

[brettkettering]

This is highly desirable, but not necessary for Secure Campaign.

[cadejager]

This is now working in pftool with a -e option. I have written jenkins tests for it and merged it into master.

Currently I am just using the fnmatch() function provided by the system. We can change this if we want different functionality.

I am going to close this issue for now. Do we need to add exclude support to the scripts?

[cadejager]

We still need to add this to the pfscripts

[jti-lanl]

Thanks. Let me me know when, and we can re-tag.

[cadejager]

I edited the pfscripts and retested.