Trying to capture protobufServer() output from PowerDNS recursor, but not seeing anything (v0.33.0)

[james-stevens]

I'm using the PowerDNS recursor to implement DNS filtering using an RPZ. I should be able to see RPZ block events logged via their protobuf interface. Looking at its internal metrics, vector is reporting that it is receiving bytes, but not logging anything to file, but I can get it to log the dnstap data from pdns_recursor fine.

https://docs.powerdns.com/recursor/lua-config/protobuf.html

PowerDNS provide the proto file (see link), which I compiled,using protoc, into a desc file and vector validate is happy with picking up the message type from the desc.

My vector config

healthchecks:
  require_healthy: true

sources:
  src_my_prom:
    type: internal_metrics

  dnstap_src:
    type: dnstap
    raw_data_only: false
    socket_path: /run/vector/dnstap.sock
    socket_file_mode: 0o777

  dns_msg:
    type: socket
    address: "0.0.0.0:4100"
    mode: "tcp"
    decoding:
      codec: protobuf
      protobuf:
        desc_file: /etc/vector/dnsmessage.desc
        message_type: PBDNSMessage


sinks:
  output_my_prom:
    type: prometheus_exporter
    address: 0.0.0.0:9598
    inputs: [ src_my_prom ]

  msg_to_files:
    type: file
    inputs: [ dns_msg ]
    healthcheck:
      enabled: true
    framing:
      method: "newline_delimited"
    path: "/opt/dnsmsg/%Y/%m/%d/%H/%M/dnsmsg_%s.log"
    encoding:
      codec: json
      metric_tag_values: full

  tap_to_files:
    type: file
    inputs: [ dnstap_src ]
    healthcheck:
      enabled: true
    framing:
      method: "newline_delimited"
    path: "/opt/dnstap/%Y/%m/%d/%H/%M/dnstap_%s.log"
    encoding:
      codec: json
      metric_tag_values: full

Then I have protobufServer("127.0.0.1:4100") in my recursor.lua to enable the pdns-recursor end.

Using netstat -nap I can see the two applications are connected to each other.

When I curl the metrics from vector, and look for vector_component_received_bytes_total & dns_msg, I can see bytes coming in, but I never see any files logged in /opt/dnsmsg (directory is owned by vector).

The metric vector_component_received_event_bytes_total for msg_to_files is also zero.

There's nothing logged by either application in syslog or systemctl

[mattb-nn]

I'm experiencing the very same issue, and I wonder if pdns-recursor isn't sending the protobuf messages over the socket framestream delimited, as with the dnstap source.

I've tried using the different framing methods available in Vector for recognising this, but didn't have any success.

I have validated that pdns-recursor is, definitely, sending the messages properly by using dmachard's Python-based protobuf receiver which outputs a representation of the messages OK.

Sorry I can't shed any light on this.

[james-stevens]

Sorry I can't shed any light on this.

Thank you so much for the feedback - very helpful - trying Python was next on my list :)

Its very odd the dnstap from recursor works fine, but not its proprietary protobuf output. Somebody suggested to me it was something to do with using different number of bytes for he buffer length - 16 vs 32 or something, but I've not looked into that.

I have validated that pdns-recursor is, definitely, sending the messages properly

However, if it works with Python & not vector, this suggests to me the issue is in vector??

Unfortunately we really need access to this output as we're using RPZs & this is the way they provide to see the times it blocks a DNS request .. and I'd really rather not have to use aa shim between the two!

[mattb-nn]

There's also this tool, again by dmachard, which handles the Protobuf messages from a pdns-recursor without any issues.

I'm not suggesting it's the best way forward, but if one had a pressing need to handle those custom protobuf messages and then pass them on to a Vector socket listener as json, then it might get you up and running.

My next debugging step is to take a pcap of the incoming pdns-recursor traffic and take a look at the structures. I, too, wonder if there is some difference in the frame size.

[james-stevens]

All we really need from the protobuf is the RPZ block logging, so we've written some LUA that allows us to output that as syslog, so passing that syslog to vector is currently our fall-back position, but I can see it being relatively compute expensive, compared to getting the protobuf output.

Thanks for the link (& all your feedback) - I'll take a look.

[johnhtodd]

I think the problem you described here a bit later in more detail: #19323

[james-stevens]

I tried a shim in C that just extends the length to 4 bytes, basically by prefixing x0000 at the front of every [length+packet], but that made no difference - so in the end I wrote a protobuf-to-JSON decoder in C and passed JSON to vector.

It's not actually that hard as protobuf-c will create all the code for handling the protobuf part and Oleg Efimov wrote a generic protobuf->JSON converter here - https://github.com/Sannis/protobuf2json-c

So I just had to stitch these two halves together.

As the shim uses protobuf-c to convert the .proto file to C code, the same shim can be easily reused to convert pretty much any protobuf to json to pass onto vector.

The nice thing about doing it this way is that it retains the schema & all the field names from the original .proto file when creating the JSON.

We're hoping to publish it as open source, but company processes are long & laborious, so I might just have to rewrite it & publish that!!!!