Additional output JSON

[McStork]

Hello,
Here is a proposal for adding an additional DSC output to the already existing XML one.

I already discussed a little of this matter on the mailing list but did not get quite a finale answer, except that some subscribers of the mailing list seem to be interested.

Basically I suggest to add an option 'add_output' that allows to generate JSON or Extended JSON (MongoDB) along with XML files.
JSON is a popular format and can be processed by many databases: MongoDB, ElasticSearch, PostgreSQL, SQL Server 2016, ...

This would allow users to have easy processing of DSC outputs for their businesses.

Code changes can be seen in PR #2. Any feedback is welcome.

[McStork]

Capture sample with 'add_output json' option set in config file:

{
  "pcap_stats": {
    "start_time": "2016-03-04T09:58:00Z",
    "stop_time": "2016-03-04T09:59:00Z",
    "dimensions": [ "ifname", "pcap_stat" ],
    "data": [
      {
        "ifname": "eth0",
        "pcap_stat": [
          { "val": "pkts_captured", "count": 588 },
          { "val": "filter_received", "count": 588 }
        ]
      }
    ]
  },
  "priming_responses": {
    "start_time": "2016-03-04T09:58:00Z",
    "stop_time": "2016-03-04T09:59:00Z",
    "dimensions": [ "All", "ReplyLen" ],
    "data": [ ]
  },
  "priming_queries": {
    "start_time": "2016-03-04T09:58:00Z",
    "stop_time": "2016-03-04T09:59:00Z",
    "dimensions": [ "Transport", "EDNSBufSiz" ],
    "data": [ ]
  },
  "qtype_total": {
    "start_time": "2016-03-04T09:58:00Z",
    "stop_time": "2016-03-04T09:59:00Z",
    "dimensions": [ "All", "Qtype" ],
    "data": [
      {
        "All": "ALL",
        "Qtype": [
          { "val": "12", "count": 6 },
          { "val": "1", "count": 4 },
          { "val": "28", "count": 1 }
        ]
      }
    ]
  },
  "qname_qtype": {
    "start_time": "2016-03-04T09:58:00Z",
    "stop_time": "2016-03-04T09:59:00Z",
    "dimensions": [ "Qname", "Qtype" ],
    "data": [
      {
        "Qname": "github.com",
        "Qtype": [
          { "val": "1", "count": 3 }
        ]
      }
    ]
  }
}

[hassaniqbal2k3]

Very useful.

[hassaniqbal2k3]

i have a need to process the dns stats and store in db for further analysis. was thinking of parsing the xml output. this json output support was much needed.

[jelu]

Thanks for the PR, I will be looking at this next week. Just recently join DNS-OARC so I will need some time to setup a development environment to test this. There are a few things I see right away and I may do a PR to your branch later on with a few changes.

[McStork]

@hassaniqbal2k3 Just being curious. What db would you plan to use?

@jelu Sure, you are welcome to open a PR.

[hassaniqbal2k3]

@McStork Cassandra is under consideration, but after looking at the extended json and mongo db combo that you have mentioned, we will study this as well.

[jelu]

Format has changed because the dataset name is not unique. Base JSON object is now an array in which a list of dataset objects resides. ISO8601 dates have been removed to conform to the XML output.

{
  "name": "pcap_stats",
  "start_time": 1457602000,
  "stop_time": 1457602020,
  "dimensions": [ "ifname", "pcap_stat" ],
  "data": [
    {
      "ifname": "eth0",
      "pcap_stat": [
        { "val": "pkts_captured", "count": 14 },
        { "val": "filter_received", "count": 14 }
      ]
    }
  ]
}

[McStork]

@jelu That looks good! Having timestamps instead of ISO8601 is also fine.