set: fix byte order metadata for host-endian types and map data

[anthonyrisinger]

Fix two issues with set/map userdata byte order metadata that cause nft list ruleset to misinterpret element keys and data values on little-endian systems. The kernel-level data is always correct — these are display/round-trip issues affecting the userdata TLVs that the nft CLI uses to reconstruct human-readable output.

Bug 1: Anonymous sets ignore explicit KeyByteOrder

Anonymous/constant/interval sets unconditionally emit KEYBYTEORDER=2 (big-endian), ignoring the KeyByteOrder field. The nft C tool always uses the key expression's actual byte order (mnl.c:mnl_nft_set_add), not a blanket default.

For host-endian types like mark_type (BYTEORDER_HOST_ENDIAN) and ifname_type (BYTEORDER_HOST_ENDIAN), this causes nft list ruleset to misinterpret the element data on LE systems:

• Mark sets: meta mark { 0x00000000, 0x01000000 } instead of { 0x00000000, 0x00000001 }
• Interface name vmaps: iifname vmap { "" : jump chain1 } instead of { "eth0" : jump chain1 }

The existing comment "Semantically useless - kept for binary compatability with nft" is incorrect — KEYBYTEORDER IS semantically meaningful for host-endian types. The nft C tool uses it in netlink_delinearize_setelem to decide whether to call mpz_switch_byteorder on element keys.

Fix: When the anonymous/constant/interval condition fires, check if KeyByteOrder is explicitly set to NativeEndian and emit 1 (host) instead of 2 (big). Falls back to the original big-endian default when KeyByteOrder is unset (zero value), preserving backwards compatibility.

Bug 2: Maps never emit DATABYTEORDER

The NFTNL_UDATA_SET_DATABYTEORDER constant exists in userdata.go but is never used. The Set struct has no DataByteOrder field, and the userdata construction has no code path that emits this TLV.

The nft C tool emits DATABYTEORDER for all datamaps:

if (set_is_datamap(set->flags))
    nftnl_udata_put_u32(udbuf, NFTNL_UDATA_SET_DATABYTEORDER,
                         set->data->byteorder);

Without it, nft list ruleset uses BYTEORDER_INVALID for map data, skips the mpz_switch_byteorder call, and displays native-endian values as byte-swapped on LE systems:

# Expected:
elements = { ... : 0x00000001 }

# Without DATABYTEORDER on LE:
elements = { ... : 0x01000000 }

Fix: Add DataByteOrder field to Set struct and emit NFTNL_UDATA_SET_DATABYTEORDER in the userdata construction for maps when set.

Discovery context

Found while debugging a WireGuard mesh overlay that uses meta mark sets and maps for zone-based connection latching. The incorrect display made it extremely difficult to diagnose whether mark values were being set correctly, since nft list ruleset showed byte-swapped values but the kernel data was actually correct (verified via nft --debug=netlink list ruleset).

Testing

Verified on linux/arm64 (little-endian) that after this patch:

• Anonymous TypeMark sets display correct values (0x00000001 not 0x01000000)
• Map data with TypeMark displays correct values
• nft --debug=netlink list ruleset shows identical kernel-level bytes before and after (no functional change)
• GOOS=linux go build . and GOOS=linux go vet . pass cleanly

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.