Skip to content

Notes about IO Loop behaviors

Jump to bottom
Arne Welzel edited this page Mar 28, 2023 · 1 revision

This page holds some notes about the current IO loop and packet processing behavior. It may have gotten stale. It is also not (yet?) a definition of how network time and the IO loop are supposed to work.

Main loop and packet dispatching

High level iteration

  • All ready IO sources are collected into the "ready" vector. Ready are those that return a zero-timeout, or have a ready FD and it's time to check FDs for readiness (this is called it's "time to poll").

  • If a packet source is available, it is unconditionally added as ready, unless it is time to poll. A PktSrc returning a zero-timeout is added as ready regardless whether it's time to poll or not. The special case is mostly about FD-based PKTSrc implementations.

  • All ready IO sources are processed.

  • When the packet source is processed and a packet is available, packet dispatching happens as follows:

    • The PktSrc implementation invokes RunState.cc:dispatch_packet(p) with the current packet.
    • Network time is updated to represent the packet's timestamp, or ignored when in the past.
    • Pending timers (if any) are expired according to the new network time. This may enqueue events for later execution into the event manager.
    • The packet is processed by packet_mgr->ProcessPacket(p) which may enqueue more events (raw_packet, new_connection, vxlan_packet, analyzer events)
    • The event manager is drained through event_mgr.Drain(), executing all queued events (two rounds).

  • Further ready IO sources are processed.

  • The event manager is drained once more (two rounds) and the loop goes back to finding ready sources.

Network time

suspend_processing() and timers

When calling suspend_processing(), timers (based on network time) stop functioning and are only scheduled/expired once continue_processing() is invoked and network time starts moving again.

continue_processing() can be invoked from a Broker raised event, input reader generated events, or from events scheduled by (external) plugins that have provide an IO source (consider a JavaScript timer expiring raising an event in Zeek script that then calls continue_processing()), or by plugins invoking Zeek functionality akin to the continue_processing() directly.

Input readers

Currently, input readers are mostly driven by the DoHeartbeat(). This is invoked via heartbeat timers that are expired based on network time. It is still possible to use input reader events to call continue_processing(), however, only for the initial DoInit() generated events.

Prominently, the output of a raw input reader executing a command can not be used to trigger continue_processing(). While the command may have been completely, it's output and completion would only be found and raised to script land through heartbeats (rather than registering FDs with the IO loop for notification).

The heartbeat approach more generally also results in the behavior that a raw input reader executing a simple command takes ~1 second to raise the event back to script land, while the command will have completed almost instantly.

        print current_time(), "one_line", s;
        Input::add_event([
                $name="raw-read-cmd",
                $source="date |",
                $reader=Input::READER_RAW,
                $ev=one_line,
                ...

1680012639.79696, zeek_init
1680012640.794374, one_line, Tue 28 Mar 2023 04:10:39 PM CEST
1680012640.794564, end_of_data, raw-read, date |

An input reader watching a file for changes similarly can not be used to trigger a continue_processing() due to this.

IO sources

Poll interval (io_poll_interval_default / io_poll_interval_live)

FD based IO sources in offline mode are only polled for readiness after processing 100 packets by default. If a trace file contains less than 100 packets, such IO sources may never be processed not happen.

When working with live interfaces, the interval defaults to 10.

With Zeek 6.0 these intervals are configurable for testing or experimentation, but likely shouldn't change in production systems.

Timeout and FD based sources

IO sources that register an FD and implement GetNextTimeout() returning a non-negative value may be processed twice during a main-loop iteration when their GetNextTimeout() method returned 0.0 and their FD was found to be ready.

https://github.com/zeek/zeek/pull/2872#discussion_r1149519828

Zero-timeout sources

Before Zeek 6.0, when multiple IO sources returned a 0.0 timeout, only one such source was considered ready and processed in the current IO loop iteration.

With Zeek 6.0, all IO sources with a zero-timeout will have Process() called on them in the same loop iteration.

Clone this wiki locally