Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nixos/packetbeat: Add basic module for packetbeat #94862

Closed
wants to merge 4,538 commits into from

Conversation

@lejonet
Copy link
Contributor

@lejonet lejonet commented Aug 7, 2020

Motivation for this change

Basic module to enable configuring and enabling packetbeat to send
packet data to elasticsearch. Modeled after the journalbeat service.

We have the packetbeat package in nixpkgs for some time, but no module, so I decided to make a basic one. I decided to make the configuration for flows and protocols their separate config values, so that if all you want to modify is general settings and outputs, you don't have to copy the entire default config and maintain it yourself.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
Copy link
Contributor

@aanderse aanderse left a comment

I have left some comments I hope you find useful. Please don't hesitate to ask if you need any clarification or help on points mentioned.

mkdir -p ${cfg.stateDir}/data
mkdir -p ${cfg.stateDir}/logs
'';
serviceConfig = {

This comment has been minimized.

@aanderse

aanderse Aug 8, 2020
Contributor

Please do not run as root if not required. Maybe DynamicUser would be appropriate.

This comment has been minimized.

@lejonet

lejonet Aug 8, 2020
Author Contributor

This is completely copied from the official unit, packetbeat needs to run as root because it captures packets from interfaces. I dunno if CAP_NET_ADMIN might be enough instead of having root, but this is how the official systemd unit does it.

'';
};

configFlows = mkOption {

This comment has been minimized.

@aanderse

aanderse Aug 8, 2020
Contributor

If this is just yaml does it even need is open option? RFC42 has been promoting a settings option which I think would benefit this module greatly.

This comment has been minimized.

@lejonet

lejonet Aug 8, 2020
Author Contributor

It is just yaml, this was a quick way of getting the module usable. My plan was to first get something usable, then refactor it to be smart too.

'';
};

configProtocols = mkOption {

This comment has been minimized.

@aanderse

aanderse Aug 8, 2020
Contributor

Would this benefit from a structured type? Maybe type = with types; attrsOf (either (bool (listOf port)));.

This comment has been minimized.

@lejonet

lejonet Aug 8, 2020
Author Contributor

Not impossible, its purely yaml it needs to generate, so a structured type would probably fit. As said in other comments, this was a quick way to getting it usable.

let
cfg = config.services.packetbeat;

packetbeatYml = pkgs.writeText "packetbeat.yml" ''

This comment has been minimized.

@aanderse

aanderse Aug 8, 2020
Contributor

See comment below about a settings option which could replace this.

This comment has been minimized.

@lejonet

lejonet Aug 8, 2020
Author Contributor

I will look into that RFC, I've added the configFile option, that short-circuits the config options in the module, and lets the user handle the configuration file as they please.

@lejonet lejonet force-pushed the lejonet:packetbeat-module branch from 83033ce to 1d33bd3 Aug 8, 2020
@aanderse
Copy link
Contributor

@aanderse aanderse commented Aug 8, 2020

A bit of a rough sketch but what about something like this?

wantedBy = [ "multi-user.target" ];
preStart = ''
mkdir -p ${cfg.stateDir}/data
mkdir -p ${cfg.stateDir}/logs

This comment has been minimized.

@aanderse

aanderse Aug 8, 2020
Contributor

Is it possible to log to stdout instead so journald can pick this up?

This comment has been minimized.

@lejonet

lejonet Aug 8, 2020
Author Contributor

There is an option to log to stderr (-e) that could be used in ExecStart

This comment has been minimized.

@lejonet

lejonet Aug 8, 2020
Author Contributor

There is an option to log to stderr that should make it possible for journald to pick it up (-e).

@lejonet
Copy link
Contributor Author

@lejonet lejonet commented Aug 8, 2020

A bit of a rough sketch but what about something like this?

Well, I've merged your suggestion and fixed it up a bit, however I cannot get it to evaluate. It would seem like there is something wrong with the lib.formats.yaml.generate call, I get the following error message:
error: value is a function while a set was expected, at /home/lejonet/sources/builddirs/lejonet-nixpkgs/nixos/modules/services/logging/packetbeat.nix:149:20
Thus I cannot test if it'll work or not, even tho I very much prefer the module to be in this way. Its also a matter of it generating a config file that packetbeat accepts.

@aanderse
Copy link
Contributor

@aanderse aanderse commented Aug 12, 2020

@lejonet I'll try to get back to this in the next few days...

@aanderse
Copy link
Contributor

@aanderse aanderse commented Aug 19, 2020

I'm sorry this took me so long to circle back to. I've pushed another copy which actually builds now... here.

Maybe @Infinisil wants to take a quick look over this module and offer some helpful advice in general, and specifically for the 2 problematic lines:

      packetbeat = {
        interfaces.device = "any";

        # FIXME: these are the 2 problem lines
        # flows = cfg.flows;
        # protocols = cfg.protocols;
      };
@Infinisil
Copy link
Member

@Infinisil Infinisil commented Aug 19, 2020

I had to correct the type from either (bool str (listOf str)) to oneOf [ boot str (listOf str) ], but I also did a refactoring using freeform modules as introduced in #82743, which really brings this module to the next level, see Infinisil@f3f2130

@aanderse
Copy link
Contributor

@aanderse aanderse commented Aug 19, 2020

Those freeform module options look great! Thanks for taking a look at this module @Infinisil.

@lejonet
Copy link
Contributor Author

@lejonet lejonet commented Aug 19, 2020

Sweet, I'll hopefully have some time this weekend to look at this and merge your changes, so that I can do the testruns of the module.

Just as a curious question, what is the actual difference between either and oneOf? Like, semantically, it would seem like they're the same, because both of them choose exactly one of a list of options, but in this case, I guess the implementations might differ in such a way that oneOf works with the freeform module suggestion and either doesn't?

@Infinisil
Copy link
Member

@Infinisil Infinisil commented Aug 19, 2020

oneOf [ a b c ] is just shorthand for either a (either b c). In your code you used either (bool str (listOf str)) which doesn't make any sense, it would have to be either bool (either str (listOf str)) instead, which I changed to oneOf [ bool str (listOf str) ]

@lejonet
Copy link
Contributor Author

@lejonet lejonet commented Aug 22, 2020

oneOf [ a b c ] is just shorthand for either a (either b c). In your code you used either (bool str (listOf str)) which doesn't make any sense, it would have to be either bool (either str (listOf str)) instead, which I changed to oneOf [ bool str (listOf str) ]

Ah that makes sense, I honestly thought that either was that shorthand, then I know

@lejonet
Copy link
Contributor Author

@lejonet lejonet commented Aug 29, 2020

I'm getting a weird issue that I hope that you guys can help with, my nix-fu is way too rusty and basic to find the cause of this. With the module in the current state and the following config:

packetbeat = {
  enable = true;
  package = pkgs.packetbeat7;
  settings = {
    name = config.networking.hostName;
    packetbeat = {
      interfaces = {
        device = "ens3";
        type = "af_packet";
        ignore_outgoing = true;
      };
      protocols = {
        tls = {
          ports = [ 443 993 995 5223 8443 8883 9243 ];
          send_certificates = true;
          include_raw_certificates = false;
          include_detailed_fields = true;
          fingerprints = [ "md5" "sha1" "sha256" ];
        };
      };
    };
    output = {
      file = {
        path = "/var/log/packetbeat";
        filename = "packetbeat.log";
        rotate_every_kb = 102400;
        number_of_files = 20;
      };
    };
    fields = {
      env = "utility";
    };
    processors = [ "add_host_metadata: ~" ];
   };
 };

The error I get when I try to deploy the config is:

building all machine configurations...
error: value is a string while a set was expected, at /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:262:25
(use '--show-trace' to show detailed location information)

The traceback isn't all to useful for me, but maybe it is for you, I think its whining about something with the ExecStart value, but that is supposed to be a string, isnt' it?
Here's the traceback:

building all machine configurations...
error: while evaluating the attribute 'buildCommand' of the derivation 'nixops-machines' at /home/lejonet/sources/builddirs/nixpkgs/pkgs/build-support/trivial-builders.nix:7:14:
while evaluating anonymous function at /nix/store/2piclzh8d93h9rfnwzjps4a31q0z07r4-nixops-1.7/share/nix/nixops/eval-machine-info.nix:371:46, called from undefined position:
while evaluating the attribute 'config.system.build.toplevel' at /home/lejonet/sources/builddirs/nixpkgs/nixos/modules/system/activation/top-level.nix:275:5:
while evaluating 'foldr' at /home/lejonet/sources/builddirs/nixpkgs/lib/lists.nix:52:20, called from /home/lejonet/sources/builddirs/nixpkgs/nixos/modules/system/activation/top-level.nix:139:12:
while evaluating 'fold'' at /home/lejonet/sources/builddirs/nixpkgs/lib/lists.nix:55:15, called from /home/lejonet/sources/builddirs/nixpkgs/lib/lists.nix:59:8:
while evaluating 'showWarnings' at /home/lejonet/sources/builddirs/nixpkgs/lib/trivial.nix:302:28, called from /home/lejonet/sources/builddirs/nixpkgs/nixos/modules/system/activation/top-level.nix:136:10:
while evaluating 'foldr' at /home/lejonet/sources/builddirs/nixpkgs/lib/lists.nix:52:20, called from /home/lejonet/sources/builddirs/nixpkgs/lib/trivial.nix:302:33:
while evaluating 'fold'' at /home/lejonet/sources/builddirs/nixpkgs/lib/lists.nix:55:15, called from /home/lejonet/sources/builddirs/nixpkgs/lib/lists.nix:59:8:
while evaluating the attribute 'warnings' at undefined position:
while evaluating anonymous function at /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:84:45, called from undefined position:
while evaluating the attribute 'value' at /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:379:9:
while evaluating the option `warnings':
while evaluating the attribute 'mergedValue' at /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:411:5:
while evaluating the attribute 'values' at /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:405:9:
while evaluating the attribute 'values' at /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:505:7:
while evaluating anonymous function at /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:391:28, called from /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:391:17:
while evaluating 'dischargeProperties' at /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:464:25, called from /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:392:62:
while evaluating the attribute 'value' at /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:277:44:
while evaluating anonymous function at /home/lejonet/sources/builddirs/nixpkgs/lib/attrsets.nix:234:10, called from undefined position:
while evaluating anonymous function at /home/lejonet/sources/builddirs/nixpkgs/nixos/modules/system/boot/systemd.nix:824:51, called from /home/lejonet/sources/builddirs/nixpkgs/lib/attrsets.nix:234:16:
while evaluating 'optional' at /home/lejonet/sources/builddirs/nixpkgs/lib/lists.nix:254:20, called from /home/lejonet/sources/builddirs/nixpkgs/nixos/modules/system/boot/systemd.nix:825:7:
while evaluating the attribute 'serviceConfig.Type' at undefined position:
while evaluating anonymous function at /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:84:45, called from undefined position:
while evaluating the attribute 'value' at /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:379:9:
while evaluating the option `systemd.services.packetbeat.serviceConfig':
while evaluating the attribute 'mergedValue' at /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:411:5:
while evaluating anonymous function at /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:413:22, called from /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:413:9:
while evaluating 'merge' at /home/lejonet/sources/builddirs/nixpkgs/lib/types.nix:292:20, called from /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:416:12:
while evaluating 'filterAttrs' at /home/lejonet/sources/builddirs/nixpkgs/lib/attrsets.nix:124:23, called from /home/lejonet/sources/builddirs/nixpkgs/lib/types.nix:293:35:
while evaluating anonymous function at /home/lejonet/sources/builddirs/nixpkgs/lib/attrsets.nix:125:29, called from /home/lejonet/sources/builddirs/nixpkgs/lib/attrsets.nix:125:18:
while evaluating anonymous function at /home/lejonet/sources/builddirs/nixpkgs/lib/types.nix:293:51, called from /home/lejonet/sources/builddirs/nixpkgs/lib/attrsets.nix:125:62:
while evaluating the attribute 'ExecStart' at /home/lejonet/sources/builddirs/nixpkgs/lib/attrsets.nix:344:7:
while evaluating anonymous function at /home/lejonet/sources/builddirs/nixpkgs/lib/types.nix:293:86, called from /home/lejonet/sources/builddirs/nixpkgs/lib/attrsets.nix:344:15:
while evaluating the attribute 'optionalValue' at /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:424:5:
while evaluating the attribute 'values' at /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:405:9:
while evaluating the attribute 'values' at /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:505:7:
while evaluating anonymous function at /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:391:28, called from /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:391:17:
while evaluating 'dischargeProperties' at /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:464:25, called from /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:392:62:
while evaluating the attribute 'value' at /home/lejonet/sources/builddirs/nixpkgs/lib/types.nix:297:60:
while evaluating the attribute 'value' of the derivation 'packetbeat.yml' at /home/lejonet/sources/builddirs/nixpkgs/pkgs/build-support/trivial-builders.nix:7:14:
while evaluating anonymous function at /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:84:45, called from undefined position:
while evaluating 'isType' at /home/lejonet/sources/builddirs/nixpkgs/lib/types.nix:14:18, called from /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:85:14:
while evaluating the attribute '_type' at undefined position:
while evaluating anonymous function at /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:280:39, called from undefined position:
while evaluating 'mergeModules'' at /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:237:36, called from /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:298:13:
while evaluating 'flip' at /home/lejonet/sources/builddirs/nixpkgs/lib/trivial.nix:138:16, called from /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:280:6:
while evaluating 'byName' at /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:260:25, called from /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:268:21:
while evaluating anonymous function at /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:261:22, called from /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:261:9:
value is a string while a set was expected, at /home/lejonet/sources/builddirs/nixpkgs/lib/modules.nix:262:25
jtojnar and others added 12 commits Sep 1, 2020
It has not been used by KDE for many years and depends on umaintained libraries we want to drop (Qt4 and Gamin).
They have been broken and disabled for ages and now the dependencies are being removed.
It is insecure and broken.

Follow #93398
it is insecure and broken
fixes at least 17 CVEs
also adds test to package
previously it was using the legacy one
ncspot: 0.2.1 -> 0.2.2
plasma5: 5.17.5 -> 5.18.5
doppler: 3.10.1 -> 3.10.3
risicle and others added 4 commits Sep 4, 2020
meshlab: 2020.03 -> 2020.07
@lejonet lejonet closed this Sep 4, 2020
@lejonet lejonet deleted the lejonet:packetbeat-module branch Sep 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.