Add a --omit-nulls option to the JSON export

[dominiklohmann]

This adds the vast.export.json.omit-nulls option that changes the JSON writer to skip over fields in JSON objects whose value is null.

Implementation-wise this turned out to be a bit more complicated than I had anticipated, as it required changes to both the JSON printer and the ostream writer. I re-used the same policy template argument for both.

📝 Checklist

• All user-facing changes have changelog entries.
• The changes are reflected on docs.tenzir.com/vast, if necessary.
• The PR description contains instructions for the reviewer, if necessary.

🎯 Review Instructions

First commit contains the code changes, which should be straightforward. The second commit contains a new integration test, which exports the same data as the test right before it, except that nulls are omitted.

Please also give feedback on the --omit-nulls naming; maybe there's a more intuitive name out there, or something that other tools already expose that we can re-use such that users may already be familiar with the flag.

Note that we should only merge this after the December release, as we're in feature freeze as of writing this.

[tobim]

I added a unit test as a tool to try to simplify the json printable part:

diff --git a/libvast/test/format/writer.cpp b/libvast/test/format/writer.cpp
index 5623b9a0f..4f364ebdd 100644
--- a/libvast/test/format/writer.cpp
+++ b/libvast/test/format/writer.cpp
@@ -32,16 +32,18 @@ auto first_csv_http_log_line = "type,ts,uid,id.orig_h,id.orig_p,id.resp_h,id.res
 auto last_csv_http_log_line = R"__(zeek.http,2009-11-19T07:17:28.829955072,"rydI6puScNa",192.168.1.104,1224,87.106.66.233,80,1,"POST","87.106.66.233","/rpc.html?e=bl",,"SCSDK-6.0.0",1064,96,200,"OK",100,"Continue",,"",,,,"application/octet-stream",,)__";
 
 auto first_zeek_conn_log_line = R"__({"ts": "2009-11-18T08:00:21.486539008", "uid": "Pii6cUUq1v4", "id.orig_h": "192.168.1.102", "id.orig_p": 68, "id.resp_h": "192.168.1.1", "id.resp_p": 67, "proto": "udp", "service": null, "duration": "163.82ms", "orig_bytes": 301, "resp_bytes": 300, "conn_state": "SF", "local_orig": null, "missed_bytes": 0, "history": "Dd", "orig_pkts": 1, "orig_ip_bytes": 329, "resp_pkts": 1, "resp_ip_bytes": 328, "tunnel_parents": []})__";
+
+auto last_zeek_http_log_line_json = R"__({"ts": "2009-11-19T07:17:28.829955072", "uid": "rydI6puScNa", "id.orig_h": "192.168.1.104", "id.orig_p": 1224, "id.resp_h": "87.106.66.233", "id.resp_p": 80, "trans_depth": 1, "method": "POST", "host": "87.106.66.233", "uri": "/rpc.html?e=bl", "user_agent": "SCSDK-6.0.0", "request_body_len": 1064, "response_body_len": 96, "status_code": 200, "status_msg": "OK", "info_code": 100, "info_msg": "Continue", "tags": [], "mime_type": "application/octet-stream"})__";
 // clang-format on
 
 template <class Writer>
-std::vector<std::string> generate(const std::vector<table_slice>& xs) {
+std::vector<std::string>
+generate(const std::vector<table_slice>& xs, caf::settings options = {}) {
   std::string str;
   caf::containerbuf<std::string> sb{str};
   auto out = std::make_unique<std::ostream>(&sb);
-  caf::settings options;
   Writer writer{std::move(out), options};
-  for (auto& x : xs)
+  for (const auto& x : xs)
     if (auto err = writer.write(x))
       FAIL("failed to write event");
   writer.flush();
@@ -64,9 +66,16 @@ TEST(CSV writer) {
   CHECK_EQUAL(lines.back(), last_csv_http_log_line);
 }
 
-TEST(JSON writer) {
+TEST(JSON writer - defaults) {
   auto lines = generate<format::json::writer>(zeek_conn_log);
   CHECK_EQUAL(lines.front(), first_zeek_conn_log_line);
 }
 
+TEST(JSON writer - omit-nulls) {
+  caf::settings options;
+  caf::put(options, "vast.export.json.omit-nulls", true);
+  auto lines = generate<format::json::writer>(zeek_http_log, options);
+  CHECK_EQUAL(lines.back(), last_zeek_http_log_line_json);
+}
+
 FIXTURE_SCOPE_END()

[dominiklohmann]

I added a unit test as a tool to try to simplify the json printable part:

@tobim Feel free to push this directly to the branch so it's properly signed. Thanks!

[tobim]

I just added some tests and tried to simplify the record to object formatting.

[dominiklohmann]

I approve of your unit tests and of the writer simplification @tobim—if you approve of my changes then I think this can be merged.

[tobim]

I looked at the implementation in detail and contributed additional tests to ensure my understanding is correct.