Emit events via JSON to capture 'no template for XXX, skip data set' so we can see data loss

[cameronkerrnz]

IPFIX-HDR:
 version=10, length=1425
 unixtime=1425336290 (4461672-07-13 09:38:42 NZST)
 seqno=2732, odid=2
[ipfix_parse_msg] no template for 262, skip data set
[ipfix_parse_msg] no template for 257, skip data set
[ipfix_parse_msg] no template for 258, skip data set
[ipfix_parse_msg] no template for 257, skip data set
[ipfix_parse_msg] no template for 258, skip data set
[ipfix_parse_msg] no template for 262, skip data set
[ipfix_parse_msg] no template for 257, skip data set
[ipfix_parse_msg] no template for 258, skip data set

This is annoying, because with default settings, you can lose a lot of data. I rather suspect that it would be useful to pre-load templates for some probes, as they will likely be the same for some devices, such as the NetScaler (ie. 258 is always an HTTP report)

[cameronkerrnz]

We would need to introduce another callback handler to notify the export modules of such an event.

This should be easy to do, the only careful bit being the naming. I'll call these 'notification' handlers, and make them type-safe. I suppose there would be scope for something more printf-like (or mlogf-like)... but that could be a more general handler; it needed at all; and since that is not clear, I won't bother with that at this time.

I'll call the handlers with the 'export_notify_no_template_for_set', and it will take the template ID (ie. setid in ipfix_parse_msg).

Possibly such a handler might try and do something with the data to prevent against data-loss, so we should pass in sufficient information to enable it to do so. Comparing with the export_drecord handler,

If we look at some code from ipfix_parse_msg, we see

        else if ( setid >255 )
        {
            /** get template
             */
            ipfixt_node_t *t;

            if ( (t=_get_ipfixt( s->templates, setid )) ==NULL ) {
                mlogf( 0, "[%s] no template for %d, skip data set\n",
                       func, setid );
                nread += setlen;
                err_flag = 1;
            }
            else {
                for ( e=g_exporter; e!=NULL; e=e->next ) {
                    if ( e->elem->export_dset )
                        (void) e->elem->export_dset( t, buf+nread, setlen,
                                                     e->elem->data );
                }

                /** read data records
                 */
                for ( offset=nread, bytesleft=setlen; bytesleft>4; ) {
                    if ( ipfix_decode_datarecord( t, buf+offset, bytesleft,
                                                  &bytes, &data ) <0 ) {
                        mlogf( 0, "[%s] set%d: decode record failed: %s\n",
                               func, i+1, strerror(errno) );
                        goto errend;
                    }

                    (void) ipfix_export_datarecord( s, t, &data, input );

                    bytesleft -= bytes;
                    offset    += bytes;
                }

                if ( bytesleft ) {
                    mlogf( 3, "[%s] set%d: skip %d bytes padding\n",
                           func, i+1, bytesleft );
                }
                nread += setlen;
            }
        }

So remembering that this callback would imply that a set or records is ignored, then we need to model this somewhat on export_dset, and not export_drecord. Below are the declarations for the export_dset callback, and the callback we are creating.

    int (*export_dset)(ipfixt_node_t*,const uint8_t*,size_t,void*);
    int (*export_notify_no_template_for_set)(int setid,ipfixs_node_t*,const uint8_t*,size_t,void*);

The ipfixs_note_t* argument is valid only when the template is found, so we don't need that. The uint8_t* and size_t will be the undecoded set contents (ie. setlen bytes from buf+nread (same as export_dset) and the final is the arg argument as commonly found in the callbacks. In short, it becomes the same as export_dset, except we substitute the unusable ipfix_node_t* argument for the template ID number. Adding the ipfixs_node_t* argument allows us to include the identity (IP) of the exporter.

[cameronkerrnz]

Example message that is currently emitted. Note that it also has the exporter's IP, to help answer the obvious question "which machines am I missing templates for?"

{ "ipfix_collector_notice":"no_template_for_set", "ipfix_template_id":"258", "ipfix_exporter_ip":"10.1.1.1", "summary":"no template for 258, skip data set", "set_bytes":""00 13 73 ...  00  01 00 01 00  01 00""}