Move policy scripts into packages

[rsmmr]

This idea has been floating around for a while: The non-standard scripts in policy/ might be better maintained as external packages these days. That way they could be updated outside of Zeek releases; and also conceptually, there isn't really that much that makes them that special compared to other packages.

We'd need to make the upgrade path smooth.
We'd probably still recommend a default set of packages to people and could keep shipping a local version of those packages as a default.

[jsiwek]

Are there more benefits to this besides being able to update them outside the normal Zeek release cycle ?

Was thinking that updates to policy/ scripts haven't been too common or critical at least recently (skimming the git logs, there's only a few changes of much significance since last release). Then also moving Zeek itself into a more frequent release schedule also makes it less an issue.

Can't think of an overall objection to doing this, just benefits seem minor.

As for implementing smoothly: could move them to git submodules so that zkg can install them independently, but then we can have the zeek distribution still include and install them by default unless user configures it not to.

[rsmmr]

The plan is to go through policy/ during the next cycles and take stock what makes sense to move.

[rsmmr]

So realistically. I don't think this is going to happen unless somebody volunteers to take on the (substantial) work. I'll remove the milestone.

[grigorescu]

Ok, this doesn't seem as bad as I thought. Some questions:

1. Where do these packages go? Are we okay with polluting github.com/zeek with a repo per package?
2. Some of these just add one or two fields to the log. If we had a good way to just toggle this functionality with a redef, we could just move the code to the base script. Right now, however, I think this would add the log fields even if the functionality to set them is not enabled.
3. What remains in Zeek? There's a category of things that would be nice to ship in the distribution, but should not be enabled (e.g. dump-events).

Here's the stuff in policy that's currently being loaded by default:

frameworks/files/detect-MHR
frameworks/files/hash-all-files
frameworks/software/version-changes
frameworks/software/vulnerable
misc/capture-loss
misc/loaded-scripts
misc/stats
policy/frameworks/notice/extend-email/hostnames
protocols/conn/known-hosts
protocols/conn/known-services
protocols/dns/detect-external-names
protocols/ftp/detect
protocols/ftp/software
protocols/http/detect-sqli
protocols/http/software
protocols/smtp/software
protocols/ssh/detect-bruteforcing
protocols/ssh/geo-data
protocols/ssh/interesting-hostnames
protocols/ssh/software
protocols/ssl/known-certs
protocols/ssl/log-hostcerts-only
protocols/ssl/validate-certs
tuning/defaults
frameworks/signatures/detect-windows-shells

More detailed comments for each script:

• files/unified2/main: The whole unified2 analyzer should be moved to a plugin, or perhaps mothballed.

• files/x509/log-ocsp: Easy. Depends on a single event, ocsp_response_certificate. This does feel a bit weird, because there's a file analyzer for this (ANALYZER_OCSP_REPLY), which would be left without much functionality in base. Perhaps once more, the whole analyzer should move.

• frameworks/control: I believe that ZeekControl is the only thing using this, so probably revisit after the Supervisor work, and remove.

• frameworks/dpd/detect-protocols: A bit tricker, but I think this would be a good candidate. I was looking at revamping this a bit anyway, as it'd be nice to detect not just protocols on non-standard ports, but also, say, a local SMB server being accessed from an external IP.

• frameworks/dpd/packet-segment-logging: Trivial

• frameworks/files/detect-MHR: Loaded by default. This does depend on hash-all-files, though perhaps that needs to be fixed. MHR doesn't have all file types anyway. Would there be any ill effects to attaching the hash analyzers twice to the same file?

• frameworks/files/{entropy-test,extract,hash}-all-files: Loaded by default. Easy. Should probably be combined into a single script, with options that control which analyzers are attached to all files.

• frameworks/intel: Tricky. I think that some of these (do_expire, do_notice, removal, whitelist) should be moved into base. seen/* is more difficult, as we need that loaded in order for intel matching to work, but loading those will slow down operations even with no intel.

• frameworks/netcontrol/catch-and-release: A lot of code, hard to test well, but mostly self-contained. frameworks/notice/actions/drop should also move into the same package.

• frameworks/notice/extend-email/hostnames: Self-contained, but loaded by default.

• frameworks/packet-filter/shunt: This one's a bit weird. All it does is provide some helper functions for BPF-based shunting. Could be loaded by default, the only penalty would be that it does have a connection_state_remove handler.

• frameworks/signatures/detect-windows-shells.sig: Loaded by default. Would be easy to move over.

• frameworks/software/version-changes: Loaded by default. Easy to move over, but perhaps due for an overhaul to Brokerize and/or persist the version data.

• frameworks/software/vulnerable: Loaded by default. Easy. This had functionality for dynamically updating this over DNS, which AFAIK was never implemented. Might be better to look at redoing this with options, to enable dynamic updates via that instead.

• frameworks/software/windows-version-detection: Feels like one of the best cases for a package.

• integration/barnyard2: I think remove this. This basically just provides an event that Broker can hit, which will log the alert. Maybe move it to community ID?

• integration/collective-intel: Self-contained. Would be easy, but perhaps it should move to the same intel package as the policy/frameworks/intel stuff, and loaded on-demand?

• misc/detect-traceroute: Makes sense, should be easy.

• misc/capture-loss: Enabled by default. I think leave in the distribution.

• misc/dump-events: This is great for debugging. I think leave in the distribution, but do not enable by default.

• misc/load-balancing: I'm not sure how much this is used. Maybe combine with packet-filter/shunt as a single package for BPF-related things?

• misc/loaded-scripts: Enabled by default. I think leave in the distribution.

• misc/profiling: Unsure. I've heard for a long time that this causes performance issues. Possible to move it over without too much difficulty.

• misc/scan: Self-contained, easy.

• misc/stats: Loaded by default. Easy.

• misc/trim-trace-file: I see this in the same light as misc/dump-events. Potentially useful as a one-off?

• misc/weird-stats: Seems easy.

• protocols/conn/known-hosts: I think move this into a Known package consisting of conn/known-hosts, conn/known-services, ssl/known-certs, tuning/track-all-assets, and allow the user to toggle one or more of those.

• protocols/conn/known-services: I think move this into a Known package consisting of conn/known-hosts, conn/known-services, ssl/known-certs, tuning/track-all-assets, and allow the user to toggle one or more of those.

• protocols/conn/mac-logging: Easy.

• protocols/conn/speculative-service: This is short, but it does set some DPD globals, dpd_match_only_beginning, dpd_late_match_stop. Nothing currently conflicts with how it's setting them, though.

• protocols/conn/vlan-logging: Easy.

• protocols/conn/weirds: Easy.

• protocols/dhcp/msg-orig: This adds a field to the log, with the messages in order. Easy.

• protocols/dhcp/software: Easy.

• protocols/dhcp/sub-opts: Again, more fields to the log. Easy.

• protocols/dns/auth-addl: Sets a couple of globals (dns_skip_all_auth and dns_skip_all_addl; no conflicts today), and adds some fields. Could add this to the base scripts with a redef as well.

• protocols/dns/detect-external-names: Loaded by default. Easy. Needs to be updated for AAAA anyway.

• protocols/dns/log-original-query-case: Adds a field. I think add this to the base script with a redef as well.

• protocols/ftp/detect-bruteforcing: Easy

• protocols/ftp/detect: Loaded by default. Easy

• protocols/ftp/software: Loaded by default. Easy

• protocols/http/detect-sqli: Loaded by default. Easy

• protocols/http/detect-webapps: This is kind of outdated, but should be easy.

• protocols/http/header-names: Adds some fields to the log. Easy.

• protocols/http/software-browser-plugins: This just does Flash. Remove?

• protocols/http/software: Loaded by default. Easy.

• protocols/http/var-extraction-cookies: Easy.

• protocols/http/var-extraction-uri: Easy.

• protocols/krb/ticket-logging: Easy.

• protocols/modbus/known-masters-slaves: This seems easy, but there's a comment about this needing a lot of work, so I'm unsure. Could maybe fit into the Known package.

• protocols/modbus/track-memmap: Easy.

• protocols/mqtt: We either move this into base or make the whole analyzer a package.

• protocols/mysql/software: Easy.

• protocols/rdp/indicate_ssl: I think just move this into the base script? I think adding a single bool field for whether or not a connection was upgraded to SSL is useful enough to just have in the log.

• protocols/smb/log-cmds: This is for debugging, so I kind of view this the same way as dump-events? Leave it, as a one-off utility?

• protocols/smtp/blocklists: Easy.

• protocols/smtp/detect-suspicious-orig: Easy

• protocols/smtp/entities-excerpt: Easy

• protocols/smtp/software: Easy, loaded by default.

• protocols/ssh/detect-bruteforcing: Loaded by default. I think this needs to be split in two. First, it provides an intel seen observation, but it also detects bruteforcing. I think part of this should just move to the Intel framework, and the other part split off as a plugin.

• protocols/ssh/geo-data: Loaded by default, easy.

• protocols/ssh/interesting-hostnames: Loaded by default, easy.

• protocols/ssh/software: Loaded by defaualt, easy.

• protocols/ssl/expiring-certs: Easy.

• protocols/ssl/expiring-certs-pem: Easy.

• protocols/ssl/heartbleed: Easy. Still relevant?

• protocols/ssl/known-certs: Loaded by default. I think move this into a Known package consisting of conn/known-hosts, conn/known-services, ssl/known-certs, tuning/track-all-assets, and allow the user to toggle one or more of those.

• protocols/ssl/log-hostcerts-only: Tricky. This does change the filter on the x509 log, so it will have ripple effects.

• protocols/ssl/notary: Easy, good fit.

• protocols/ssl/validate-certs: Probably combine certs, ocsp, sct into a single package.

• protocols/ssl/validate-ocsp: Probably combine certs, ocsp, sct into a single package.

• protocols/ssl/validate-sct: Probably combine certs, ocsp, sct into a single package.

• protocols/ssl/weak-keys: Easy.

• tuning/json-logs: This is just a short-hand for a single redef. I think just leave this in the distribution, not loaded by default.

• tuning/track-all-assets: I think move this into a Known package consisting of conn/known-hosts, conn/known-services, ssl/known-certs, tuning/track-all-assets, and allow the user to toggle one or more of those.

• tuning/defaults/extracted_file_limits: Loaded by default. Sets FileExtract::default_limit to 100 MB. I think we should just set this as the default, and provide instructions for increasing it.

• tuning/defaults/packet-fragments: Loaded by default. Shortens frag_timeout to 5 minutes. I think we should just look at setting the default timeout to 5 minutes, since the current default of "never timeout" doesn't seem reasonable to me.

• tuning/defaults/warnings: Loaded by default. Emits a warning if no local nets have been defined. Maybe just make this part of base/utils/site?

[grigorescu]

Another open question here is the documentation. I have some Zeekygen + Sphinx scripts which will generate the documentation and link to the RTD docs for the core features but, for instance, new notice types don't get documented in the package docs. Right now, we document base/frameworks/notice with stuff like:

TeamCymruMalwareHashRegistry::Match (present if policy/frameworks/files/detect-MHR.zeek is loaded)

What I'd like to see is documentation for MHR which just shows which Notice types that are added by that script.

Maybe a separate issue for that?

[rsmmr]

Great list, thanks, Vlad! Yes, there are a number of conceptual questions that we'll need to answer first to make this work. The technical part of actually moving is probably the smallest piece of the work.

Some quick thoughts:

Where do these packages go? Are we okay with polluting github.com/zeek with a repo per package?

I keep mulling over this one as well. One repo per package would of course work, and may just be fine; probably also depending on number of packages we end up with. We could alternatively create a single monorepo containing everything we maintain (say, zeek-policy), and people would always install the whole thing (that might be a good thing or bad thing, not sure). Or we could create a separate GitHub account just for collecting "official" packages.

Some of these just add one or two fields to the log. If we had a good way to just toggle this functionality with a redef, we could just move the code to the base script. Right now, however, I think this would add the log fields even if the functionality to set them is not enabled.

I do think this is an opportunity to review what's in base as well, so making changes to standards logs there seems fine to me where it makes sense.

What remains in Zeek? There's a category of things that would be nice to ship in the distribution, but should not be enabled (e.g. dump-events).

Yeah, I can see leaving some technical helper scripts in there. Your list goes into the right direction I would say: everything that's clearly a self-contained unit of typical add-on analysis functionality should move into a package.

Pinging @ckreibich who might have some thoughts here as well.

[ckreibich]

To me a new GitHub account/organization feels right. Given that folks currently @load select bits of the policy tree as needed, it seems appropriate to let them load corresponding packages individually in the future (with the possibility of a higher-level collection dependency tree, as discussed on the mailing list, to simplify installation).

One thought: in the past we've repeatedly encountered a need for a stronger "standard library" for the scripting layer — something that might include additional container data types, statistical tooling, etc. I'm not sure "policy" alone captures this, so perhaps this is a good opportunity to establish a "Zeek Standard Library" that this new account would then represent.

[J-Gras]

One thought: in the past we've repeatedly encountered a need for a stronger "standard library" for the scripting layer — something that might include additional container data types, statistical tooling, etc. I'm not sure "policy" alone captures this, so perhaps this is a good opportunity to establish a "Zeek Standard Library" that this new account would then represent.

I really like the idea of a ZSL. However, I think it would be a bit strange to mix data types and detection scripts like heartbleed. Maybe the ZSL and the policy/standard scripts are two different things?

frameworks/intel: Tricky. I think that some of these (do_expire, do_notice, removal, whitelist) should be moved into base. seen/* is more difficult, as we need that loaded in order for intel matching to work, but loading those will slow down operations even with no intel.

Because that's the only one that is marked as tricky: I think that would be a candidate to revisit base as Robin already suggested. Moving the Intel Framework as a whole into its own package should be quite straight forward. I have a POC somewhere.

[anthonykasza]

If effort is going into packaging policy scripts, it may be worthwhile to consider reorganizing "utils" and "misc" scripts into packages too.

[ckreibich]

Yeah Anthony, agreed ... there's a lot of scope. But utils and misc are trickier than the policy scripts because they reside in base, so are used broadly by the main distribution. I think the policy cleanup will inform a lot of our future steps in making all this more modular.