The absence of records in Broker's data model

[ckreibich]

Recent discussions around backward-compatibility of changes to records flagged how particularly exposed users of Broker's Python bindings are to such changes. Mapping Zeek's records to Broker's data model expects records as a tuple of values, with each member in its own Broker-model rendering. The vector must cover the full list of fields, including Nones for optional ones. As a result, any change to the record definition means you must adapt your Python implementation to reflect it. This is true for both the traditional wire format and the new WebSocket one.

I am wondering whether we should extend Broker's data model to allow record types, with a serialization as a key/val store.

Does anyone recall the historical reasoning for stopping short of explicit record support in the model?

@J-Gras, @Chukudubem, @Neverlord, @rsmmr, fyi

[vpax]

I'm wondering about efficiency tradeoffs, and in particular whether there's a way to support an initial phase that agrees on the expected positions, plus support for gaps, such that the common case of sending around a large number of agreed-upon records can proceed with no need to do per-record field lookups.

[ckreibich]

It's all good Dominik! I think the performance question is interesting, but I don't really mean to change record rendering everywhere. I'm just looking to loosen the strictness of rendering Zeek records.

Let's say we don't change Broker at all. We could change Zeek so that it supports records not just in form of Broker vectors, but alternatively also as Broker tables. Nothing changes about Zeek clusters or performance because it all keeps running as-is. But for use cases like the Python bindings, where some freedom in the rendering is desirable, it would help:

When receiving a table-based record, the Python side could just grab the fields it cares about. If a field got dropped, it's not there, which can be handled better than a vector with changing lengths or member meanings.

When sending a table-based record, the Python side doesn't need to know the full set of fields. Zeek may determine that the received data is insufficient to instantiate a record because fields are missing, but that too can be handled robustly.

One catch is that in order for the Python end to receive table-based records, the other side would need to learn somehow that it should use tables. It would be nice if as part of the connection set-up the endpoints could send some config info — a config table, say. That would be nice for other reasons too: for example, it'd spare the Cluster framework from building up a table of Broker IDs to node names — the endpoint could just send that along at the very beginning.

Does that help?

[awelzel]

One catch is that in order for the Python end to receive table-based records, the other side would need to learn somehow that it should use tables. It would be nice if as part of the connection set-up the endpoints could send some config info

(considering websocket part and external client usage of this only)

IMO "table-based records" should be/have been the default. Maybe on a /v2. It could also be opt-in via query parameter in the URI or HTTP header in the initial request on /v1. Possibly could also be done backwards compatible by stuffing @field-name elements into the elements of a "record vector" so it looks like below:

      "@data-type": "vector",
          "data": [
            {
              "@field-name": "x",
              "@data-type": "count",
              "data": 42
            },
            {
              "@field-name": "y",
              "@data-type": "count",
              "data": 4711
            }
          ]

It's then implicit that vectors with @field-names are actually records, but knowledgeable clients can do things better upon receiving these. The explicit "record" type might just be as follows (and somewhat closer/usual to how a HTTP/JSON API would be).

      "@data-type": "record",
          "data": {
            "x": {
              "@data-type": "count",
              "data": 42
            },
            "y": {
              "@data-type": "count",
              "data": 4711
            }
          }

Many systems seem fine consuming conn.log, ssl.log, ... in JSON format, so not sure about the performance/optimization. Specifically for the websocket data representation: it's already verbose JSON and that record optimization seems to actually hinder broader/easier integration and adoption.

[ckreibich]

(considering websocket part and external client usage of this only)

I think that's actually right on the mark — I like the idea of focusing this on the WebSocket transport. It's where we're going for the Python bindings anyway, users of this transport are arguably already in the "comfort, not performance" class, and via the WebSocket framing we get nice versioning support. If you have a Zeek-external use case that needs an efficient transport, it seems reasonable to me to require linking with Broker.

My understanding of Dubem's use case and Robin's zeek-agent-v2 is that event rates wouldn't be a concern, but holler if I'm off there.

[ckreibich]

via the WebSocket framing we get nice versioning support

Oops, that confused Broker vs. Zeek's use of it. We'd want some way to convey the Zeek-level dialect the endpoint speaks, so I'm back to wanting some way to express configurability at the start.

stuffing @field-name elements

Could do, yeah, though all in all an explicit switch to a table seems nicer.

[Neverlord]

We probably need to define a mapping to broker::table in any case. That "trick" with adding extra annotations to a vector won't work there. The native format is also versioned, but we would need to have a mapping from v1 to v2 to make sure we can mix v1 and v2 endpoints in the network. Otherwise, we might lose information. On the /v1 WebSocket endpoints, the records would then also show up as a table.

The table representation of a record could just have __META-TYPE__ set to __RECORD__ (__ prefix+suffix is just an idea to have some sort of reserved "namespace" for such things).

On v2, these could then get flattened into the new, more efficient record data type. If we go down that route, I think we should really think through the types in Broker to see if we have actual good representation for all Zeek types or if we could use some extra types. I'd rather add a couple more types with v2 than having to do a v3 soon after.

Or, we just use the tables with some special "marker" field and see whether that's enough? May not be pretty, but we could still compress them at some point in the future with a dedicated type if performance turns out to be an issue with that representation. That would then boil down to a Zeek-side change only for now. For the performance impact, we could build a simple benchmark that renders some common record types in Zeek in both versions and then sees how many of these we can push through Broker per second.

Of course it's also an option to say the native wire-format v2 is incompatible with v1. However, we have to have the mapping between v1 and v2 in any case if we still want to offer the v1 API to WebSocket clients. Would feel weird to omit that mapping for the native format.

[rsmmr]

Does anyone recall the historical reasoning for stopping short of explicit record support in the model?

Yeah, performance indeed I think, maybe also ease of implementation.

(Also, the other notable piece missing is typing, but that applies to Broker values generally.)

[ckreibich]

Okay ... I recently learned a few things about Broker. Might well be old news for most, but here's where I land as a result:

• We are discussing two layers — Broker with its native format & WebSockets, and Zeek's internal mapping of its types onto Broker's data model (as per src/broker/Data.cc). The latter isn't versioned, based on the assumption that in a Zeek cluster everything runs the same Zeek version.

• There's no explicit typing — there's always going to be some node-local assumption of what a given type looks like as it's read/written, with the potential of other nodes disagreeing. (That's unless we sent type descriptions etc — I'll ignore this for the moment.)

• There's (obviously) Zeek-level event handling for communication purposes: a received event gets explicitly forwarded from one topic to another, possibly repackaged, perhaps responded to on another topic, RPC-style, etc — all ad-hoc. (One of the "Big Ideas" items considers formalizing these communication models.)

• There's also Broker-level message handling: Broker supports internal message relay across nodes on a given topic, all below the Zeek layer. We don't use this in Zeek: the script-level Broker::forward_messages = F disables it. This was the gist of gateways and relay of messages for zeek-agent we discussed in the past.

• Ignoring for the moment the fact that it's disabled, this property means that the notion of one node (a Python client) telling a neighbor (a Zeek cluster node) at the Zeek level that it'd like to receive records in a certain serialization format doesn't fit: the creator of an event might be somewhere else in the cluster, with no idea of the local preference. So in order to express a potential serialization preference from one node to its neighbor, we'd have to focus on the Broker level, which feels pretty bad — that's building more and more magic into Broker.

• Seems there's agreement that the cleanest way forward would be for records to be rendered as Broker tables (or a new explicit record type), which implies performance questions.

I don't really like the idea of tweaking the current Broker vectors to somehow better accommodate Zeek's records — Broker's data model should fit naturally. At least, I don't think we should do this next. My suggestion is to actually examine the impact of a switch to table-based records in Zeek. This doesn't strike me as that much work: it looks like a rewrite of a few components of src/broker/Data.{cc,h}, plus some benchmarking (say via os-perf or custom benchmark code). It's interesting and informs next steps.

I also don't think this is especially urgent given how much else is going on right now. The situation for Python / non-Zeek endpoints isn't great, and that alone was worth highlighting. But it's also not a showstopper.

Thoughts welcome...