NTP Extension Fields and MAC

This is the most advanced section of the NTP example. NTP packets can optionally include extension fields (variable-length, zero or more) and a MAC (message authentication code). Parsing these requires new combinators: h_put_value, h_free_value, h_action, h_length_value, and h_many.

Source file: ntp.c, lines 34-47

Hammer concepts: h_put_value · h_free_value · h_action · h_length_value · h_many

---

What Are Extension Fields?

After the 48 mandatory bytes (essential fields), an NTP packet may contain zero or more extension fields. Each has this structure:

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|          Field Type           |            Length             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
.                                                               .
.                       Value (variable)                        .
.                                                               .
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

The Length field gives the total size of the extension field including the 4-byte header (Field Type + Length). So the actual value data is Length - 4 bytes.

This "length includes its own header" pattern is extremely common in binary protocols.

---

The Challenge

We need to:

1. Parse the Field Type (16 bits) - easy.
2. Parse the Length (16 bits) - easy.
3. Use that Length value to know how many bytes of Value data to read - hard.

Step 3 is hard because the length of the Value depends on a previously parsed value. Standard combinators like h_sequence can't express this since each sub-parser is independent. We need data-dependent parsing.

---

Building the Extension Field Parser, Step by Step

Step 1: Parse the Field Type

Nothing new here:

H_RULE(field_type, h_int16());  // 16 bits

Step 2: Parse and Store the Length with h_put_value

We need to parse the 16-bit length and save its value for later use. h_put_value does both.

h_put_value(parser, name)

Parameter	Type	Description
parser	HParser *	The parser to run
name	const char *	A string key to store the result under

This parses normally and stores the result in a named "slot" that other combinators can retrieve later.

h_put_value(h_uint16(), "opt_len_val")

This reads a 16-bit unsigned integer and stores it under "opt_len_val".

Step 3: Retrieve the Length with h_free_value

To get the stored value back:

h_free_value(name)

Parameter	Type	Description
name	const char *	The key that was passed to h_put_value

This creates a parser that produces the previously stored value and removes it from the store. The "free" means the binding is cleaned up after use. This is important when parsing multiple extension fields, since each one reuses the same name "opt_len_val".

h_free_value("opt_len_val")   // Produces the stored uint16 value

Step 4: Transform the Length with h_action

The stored length includes the 4-byte header (Field Type + Length), but we only want to read Length - 4 bytes of value data. We need to subtract 4.

h_action(parser, action_fn, user_data)

Parameter	Type	Description
parser	HParser *	The parser whose result will be transformed
action_fn	HParsedToken *(*)(const HParseResult *, void *)	A C function that transforms the parse result
user_data	void *	Optional data passed to action_fn (usually NULL)

The action function receives the parse result and returns a new HParsedToken. From ntp.c:

HParsedToken *opt_ext_len(const HParseResult *p, void *user_data) {
    unsigned int temp = p->ast->uint;
    return H_MAKE_UINT(temp - 4);
}

It reads the integer value, subtracts 4, and returns a new unsigned integer token.

Now we wrap retrieval and transformation together:

h_action(h_free_value("opt_len_val"), opt_ext_len, NULL)

This produces a parser that yields the adjusted length.

Step 5: Read Variable-Length Data with h_length_value

Finally, we use the adjusted length to read exactly that many bytes.

h_length_value(length_parser, value_parser)

Parameter	Type	Description
length_parser	HParser *	A parser that produces an integer n
value_parser	HParser *	A parser to run n times

h_length_value runs length_parser to get a count, then runs value_parser that many times.

h_length_value(
    h_action(h_free_value("opt_len_val"), opt_ext_len, NULL),  // -> n
    h_uint8()                                                   // run n times
)

If the Length field was 0x0008 (8), then opt_ext_len computes 8 - 4 = 4, and h_length_value reads 4 bytes (four h_uint8() calls).

Step 6: Assemble the Extension Field Rule

Putting it all together:

H_RULE(ext_field,
       h_sequence(
           field_type,
           h_put_value(h_uint16(), "opt_len_val"),
           h_length_value(
               h_action(h_free_value("opt_len_val"), opt_ext_len, NULL),
               h_uint8()),
           NULL));

The data flow:

Input bytes:   [Field Type: 2B] [Length: 2B] [Value: (Length-4) bytes]
                     │               │                    │
                field_type     h_put_value ──store──►     │
                                     │                    │
                               h_free_value ◄──retrieve── │
                                     │                    │
                               h_action (subtract 4)      │
                                     │                    │
                               h_length_value ────────► read N bytes

---

Repeating Extension Fields with h_many

An NTP packet can have zero or more extension fields. h_many handles this:

h_many(parser)

Matches parser zero or more times, collecting results into a sequence. It's greedy: it keeps matching until parser fails, then succeeds with however many matches it found.

H_RULE(ext_fields, h_many(ext_field));

If there are no extension fields, ext_fields succeeds with an empty sequence. If there are three, it produces a sequence of three parsed extension fields.

Why h_free_value matters: Each extension field stores its length under "opt_len_val". h_free_value cleans up the binding after each use, so the next extension field can reuse the same name. Without it, the second extension field would see a stale value.

---

The MAC (Message Authentication Code)

After the extension fields, the packet may end with an optional MAC:

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                        Key Identifier                         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
|                     Message Digest (128 bits)                 |
|                                                               |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

The MAC is a fixed-size structure:

H_RULE(key_id, h_uint32());                            // 32 bits
H_RULE(dgst, h_sequence(h_int64(), h_int64(), NULL));  // 128 bits
H_RULE(mac, h_sequence(key_id, dgst, NULL));

The 128-bit message digest is read as two 64-bit integers (same splitting technique from timestamps).

---

Summary

Combinator	What It Does	Why We Need It
h_put_value(p, name)	Parse with p, store result as name	Save the length field for later
h_free_value(name)	Retrieve and remove a stored value	Get the length back (and clean up for reuse)
h_action(p, fn, data)	Parse with p, transform result with fn	Subtract 4 from the length
h_length_value(len, p)	Use len's result as a repeat count for p	Read exactly Length - 4 bytes
h_many(p)	Match p zero or more times	Handle variable number of extension fields

---

Next: Assembling the Parser - Combine everything with h_choice and h_left.

Previous: Parsing Data Fields