Resource data indexing and documentation

- Implement resource data index (optional)
- Implement resource data reverse index (optional)
- Add more CDS extended options
- Update documentation and man-page

CBOR_DNS_STREAM.md

@@ -373,8 +373,12 @@ An array mixed with resource data and compressed names.
 Each option is specified here as OptionName(OptionNumber) and optional
 OptionValue type.
 
-- `RLABELS(0) uint`: Indicates how many labels should be stored in the reverse index before discarding them
+- `RLABELS(0) uint`: Indicates how many labels should be stored in the reverse label index before discarding them
 - `RLABEL_MIN_SIZE(1) uint`: The minimum size a label must be to be put in the reverse label index
+- `RDATA_RINDEX_SIZE(2) uint`: Indicates how many rdata should be stored in the reverse rdata index before discarding them
+- `RDATA_RINDEX_MIN_SIZE(3) uint`: The minimum size a rdata must be to be put in the reverse rdata index
+- `USE_RDATA_INDEX(4)`: If present then the stream uses rdata indexing
+- `RDATA_INDEX_MIN_SIZE(5) uint`: The minimum size a rdata must be to be put in the rdata index
 
 ## Deduplication
 

README.md

@@ -107,14 +107,41 @@ Also note that we have observed significant memory leaks on FreeBSD
 1. reinstall the libbind port
 1. recompile and install dnscap
 
+## CBOR DNS Stream Format
+
+This is an experimental format for representing DNS information in CBOR
+with the goals to:
+- Be able to stream the information
+- Support incomplete, broken and/or invalid DNS
+- Have close to no data quality and signature degradation
+- Support additional non-DNS meta data (such as ICMP/TCP attributes)
+
+Read [CBOR_DNS_STREAM.md](https://github.com/DNS-OARC/dnscap/blob/develop/CBOR_DNS_STREAM.md) for more information.
+
+To enable this output please follow the instructions below for Enabling
+CBOR Output, note that this only requires Tinycbor.
+
+### Outputting to CBOR DNS Stream (CDS)
+
+To output to the CDS format you tell `dnscap` to write to a file and set
+the format to CDS.  CDS is a stream of CBOR objects and you can control how
+many objects are kept in memory until flushed to the file by setting
+`cds_cbor_size`, note that this is bytes of memory and not number of objects.
+When it reaches this limit it will write the output and start on a new file.
+Read `dnscap`'s man page for all CDS extended options.
+
+```
+src/dnscap [...] -w <file> -F cds [ -o cds_cbor_size=<bytes> ]
+```
+
 ## CBOR
 
 There is experimental support for CBOR output using LDNS and Tinycbor with
 a data structure described in the DNS-in-JSON draft.
 
 https://datatracker.ietf.org/doc/draft-hoffman-dns-in-json/
 
-### Enabling CBOR output
+### Enabling CBOR Output
 
 To enable the CBOR output support you will need to install it's dependencies
 before running `configure`, LDNS exists for most distributions but Tinycbor
@@ -151,7 +178,7 @@ new file.  You can control the number of bytes with the extended option
 `cbor_chunk_size`.
 
 ```
-src/dnscap -w <file> -F cbor [ -o cbor_chunk_size=<bytes>]
+src/dnscap [...] -w <file> -F cbor [ -o cbor_chunk_size=<bytes> ]
 ```
 
 ### Additional attributes