SQL: Add type headers to response formats.

docs/querying/sql.md

@@ -879,6 +879,8 @@ Submit your query as the value of a "query" field in the JSON object within the
 |`query`|SQL query string.| none (required)|
 |`resultFormat`|Format of query results. See [Responses](#responses) for details.|`"object"`|
 |`header`|Whether or not to include a header row for the query result. See [Responses](#responses) for details.|`false`|
+|`typesHeader`|Whether or not to include type information in the header. Can only be set when `header` is also `true`. See [Responses](#responses) for details.|`false`|
+|`sqlTypesHeader`|Whether or not to include SQL type information in the header. Can only be set when `header` is also `true`. See [Responses](#responses) for details.|`false`|
 |`context`|JSON object containing [connection context parameters](#connection-context).|`{}` (empty)|
 |`parameters`|List of query parameters for parameterized queries. Each parameter in the list should be a JSON object like `{"type": "VARCHAR", "value": "foo"}`. The type should be a SQL type; see [Data types](#data-types) for a list of supported SQL types.|`[]` (empty)|
 
@@ -920,44 +922,60 @@ Metadata is available over HTTP POST by querying [metadata tables](#metadata-tab
 
 #### Responses
 
+##### Result formats
+
 Druid SQL's HTTP POST API supports a variety of result formats. You can specify these by adding a "resultFormat"
 parameter, like:
 
 ```json
 {
   "query" : "SELECT COUNT(*) FROM data_source WHERE foo = 'bar' AND __time > TIMESTAMP '2000-01-01 00:00:00'",
-  "resultFormat" : "object"
+  "resultFormat" : "array"
 }
 ```
 
-The supported result formats are:
-
-|Format|Description|Content-Type|
-|------|-----------|------------|
-|`object`|The default, a JSON array of JSON objects. Each object's field names match the columns returned by the SQL query, and are provided in the same order as the SQL query.|application/json|
-|`array`|JSON array of JSON arrays. Each inner array has elements matching the columns returned by the SQL query, in order.|application/json|
-|`objectLines`|Like "object", but the JSON objects are separated by newlines instead of being wrapped in a JSON array. This can make it easier to parse the entire response set as a stream, if you do not have ready access to a streaming JSON parser. To make it possible to detect a truncated response, this format includes a trailer of one blank line.|text/plain|
-|`arrayLines`|Like "array", but the JSON arrays are separated by newlines instead of being wrapped in a JSON array. This can make it easier to parse the entire response set as a stream, if you do not have ready access to a streaming JSON parser. To make it possible to detect a truncated response, this format includes a trailer of one blank line.|text/plain|
-|`csv`|Comma-separated values, with one row per line. Individual field values may be escaped by being surrounded in double quotes. If double quotes appear in a field value, they will be escaped by replacing them with double-double-quotes like `""this""`. To make it possible to detect a truncated response, this format includes a trailer of one blank line.|text/csv|
-
-You can additionally request a header by setting "header" to true in your request, like:
+You can additionally request a header with information about column names by setting `header` to true in your request.
+When you set `header` to true, you can optionally include `typesHeader` and `sqlTypesHeader` as well, which gives
+you information about [Druid runtime and SQL types](#data-types) respectively. You can request all these headers
+with a request like:
 
 ```json
 {
   "query" : "SELECT COUNT(*) FROM data_source WHERE foo = 'bar' AND __time > TIMESTAMP '2000-01-01 00:00:00'",
-  "resultFormat" : "arrayLines",
-  "header" : true
+  "resultFormat" : "array",
+  "header" : true,
+  "typesHeader" : true,
+  "sqlTypesHeader" : true
 }
 ```
 
-In this case, the first result of the response body is the header row. For the `csv`, `array`, and `arrayLines` formats, the header
-will be a list of column names. For the `object` and `objectLines` formats, the header will be an object where the
-keys are column names, and the values are null.
+The supported result formats are:
+
+|Format|Description|Header description|Content-Type|
+|------|-----------|------------------|------------|
+|`object`|The default, a JSON array of JSON objects. Each object's field names match the columns returned by the SQL query, and are provided in the same order as the SQL query.|If `header` is true, the first row is an object where the fields are column names. Each field's value is either null (if `typesHeader` and `sqlTypesHeader` are false) or an object that contains the Druid type as `type` (if `typesHeader` is true) and the SQL type as `sqlType` (if `sqlTypesHeader` is true).|application/json|
+|`array`|JSON array of JSON arrays. Each inner array has elements matching the columns returned by the SQL query, in order.|If `header` is true, the first row is an array of column names. If `typesHeader` is true, the next row is an array of Druid types. If `sqlTypesHeader` is true, the next row is an array of SQL types.|application/json|
+|`objectLines`|Like "object", but the JSON objects are separated by newlines instead of being wrapped in a JSON array. This can make it easier to parse the entire response set as a stream, if you do not have ready access to a streaming JSON parser. To make it possible to detect a truncated response, this format includes a trailer of one blank line.|Same as "object".|text/plain|
+|`arrayLines`|Like "array", but the JSON arrays are separated by newlines instead of being wrapped in a JSON array. This can make it easier to parse the entire response set as a stream, if you do not have ready access to a streaming JSON parser. To make it possible to detect a truncated response, this format includes a trailer of one blank line.|Same as "array", except the rows are separated by newlines.|text/plain|
+|`csv`|Comma-separated values, with one row per line. Individual field values may be escaped by being surrounded in double quotes. If double quotes appear in a field value, they will be escaped by replacing them with double-double-quotes like `""this""`. To make it possible to detect a truncated response, this format includes a trailer of one blank line.|Same as "array", except the lists are in CSV format.|text/csv|
+
+If `typesHeader` is set to true, [Druid type](#data-types) information is included in the response. Complex types,
+like sketches, will be reported as `COMPLEX<typeName>` if a particular complex type name is known for that field,
+or as `COMPLEX` if the particular type name is unknown or mixed. If `sqlTypesHeader` is set to true,
+[SQL type](#data-types) information is included in the response. It is possible to set both `typesHeader` and
+`sqlTypesHeader` at once. Both parameters require that `header` is also set.
+
+To aid in building clients that are compatible with older Druid versions, Druid returns the HTTP header
+`X-Druid-SQL-Header-Included: yes` if `header` was set to true and if the version of Druid the client is connected to
+understands the `typesHeader` and `sqlTypesHeader` parameters. This HTTP response header is present irrespective of
+whether `typesHeader` or `sqlTypesHeader` are set or not.
 
 Druid returns the SQL query identifier in the `X-Druid-SQL-Query-Id` HTTP header.
 This query id will be assigned the value of `sqlQueryId` from the [connection context parameters](#connection-context)
 if specified, else Druid will generate a SQL query id for you.
 
+##### Errors
+
 Errors that occur before the response body is sent will be reported in JSON, with an HTTP 500 status code, in the
 same format as [native Druid query errors](../querying/querying.md#query-errors). If an error occurs while the response body is
 being sent, at that point it is too late to change the HTTP status code or report a JSON error, so the response will
@@ -969,8 +987,9 @@ trailer they all include: one blank line at the end of the result set. If you de
 through a JSON parsing error or through a missing trailing newline, you should assume the response was not fully
 delivered due to an error.
 
-### HTTP DELETE
-You can use the HTTP `DELETE` method to cancel a SQL query on either the Router or the Broker. When you cancel a query, Druid handles the cancellation in a best-effort manner. It marks the query canceled immediately and aborts the query execution as soon as possible. However, your query may run for a short time after your cancellation request.
+### Cancelling queries
+
+You can use the HTTP DELETE method to cancel a SQL query on either the Router or the Broker. When you cancel a query, Druid handles the cancellation in a best-effort manner. It marks the query canceled immediately and aborts the query execution as soon as possible. However, your query may run for a short time after your cancellation request.
 
 Druid SQL's HTTP DELETE method uses the following syntax:
 ```

integration-tests/src/test/java/org/apache/druid/tests/query/ITSqlCancelTest.java

@@ -88,7 +88,7 @@ public void testCancelValidQuery() throws Exception
       queryResponseFutures.add(
           sqlClient.queryAsync(
               sqlHelper.getQueryURL(config.getRouterUrl()),
-              new SqlQuery(QUERY, null, false, ImmutableMap.of("sqlQueryId", queryId), null)
+              new SqlQuery(QUERY, null, false, false, false, ImmutableMap.of("sqlQueryId", queryId), null)
           )
       );
     }
@@ -125,7 +125,7 @@ public void testCancelInvalidQuery() throws Exception
     final Future<StatusResponseHolder> queryResponseFuture = sqlClient
         .queryAsync(
             sqlHelper.getQueryURL(config.getRouterUrl()),
-            new SqlQuery(QUERY, null, false, ImmutableMap.of("sqlQueryId", "validId"), null)
+            new SqlQuery(QUERY, null, false, false, false, ImmutableMap.of("sqlQueryId", "validId"), null)
         );
 
     // Wait until the sqlLifecycle is authorized and registered

services/src/test/java/org/apache/druid/server/AsyncQueryForwardingServletTest.java

@@ -205,7 +205,7 @@ public void testDeleteBroadcast() throws Exception
   @Test
   public void testSqlQueryProxy() throws Exception
   {
-    final SqlQuery query = new SqlQuery("SELECT * FROM foo", ResultFormat.ARRAY, false, null, null);
+    final SqlQuery query = new SqlQuery("SELECT * FROM foo", ResultFormat.ARRAY, false, false, false, null, null);
     final QueryHostFinder hostFinder = EasyMock.createMock(QueryHostFinder.class);
     EasyMock.expect(hostFinder.findServerSql(query))
             .andReturn(new TestServer("http", "1.2.3.4", 9999)).once();

services/src/test/java/org/apache/druid/server/router/ManualTieredBrokerSelectorStrategyTest.java

@@ -247,6 +247,8 @@ private SqlQuery createSqlQueryWithContext(Map<String, Object> queryContext)
         "SELECT * FROM test",
         null,
         false,
+        false,
+        false,
         queryContext,
         null
     );

services/src/test/java/org/apache/druid/server/router/TieredBrokerHostSelectorTest.java

@@ -391,6 +391,8 @@ private SqlQuery createSqlQueryWithContext(Map<String, Object> queryContext)
         "SELECT * FROM test",
         null,
         false,
+        false,
+        false,
         queryContext,
         null
     );

sql/src/main/java/org/apache/druid/sql/SqlRowTransformer.java

@@ -36,6 +36,7 @@
 public class SqlRowTransformer
 {
   private final DateTimeZone timeZone;
+  private final RelDataType rowType;
   private final List<String> fieldList;
 
   // Remember which columns are time-typed, so we can emit ISO8601 instead of millis values.
@@ -45,6 +46,7 @@ public class SqlRowTransformer
   SqlRowTransformer(DateTimeZone timeZone, RelDataType rowType)
   {
     this.timeZone = timeZone;
+    this.rowType = rowType;
     this.fieldList = new ArrayList<>(rowType.getFieldCount());
     this.timeColumns = new boolean[rowType.getFieldCount()];
     this.dateColumns = new boolean[rowType.getFieldCount()];
@@ -56,6 +58,11 @@ public class SqlRowTransformer
     }
   }
 
+  public RelDataType getRowType()
+  {
+    return rowType;
+  }
+
   public List<String> getFieldList()
   {
     return fieldList;

sql/src/main/java/org/apache/druid/sql/http/ArrayLinesWriter.java

@@ -22,11 +22,11 @@
 import com.fasterxml.jackson.core.JsonGenerator;
 import com.fasterxml.jackson.core.io.SerializedString;
 import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.calcite.rel.type.RelDataType;
 
 import javax.annotation.Nullable;
 import java.io.IOException;
 import java.io.OutputStream;
-import java.util.List;
 
 public class ArrayLinesWriter implements ResultFormat.Writer
 {
@@ -57,15 +57,13 @@ public void writeResponseEnd() throws IOException
   }
 
   @Override
-  public void writeHeader(final List<String> columnNames) throws IOException
+  public void writeHeader(
+      final RelDataType rowType,
+      final boolean includeTypes,
+      final boolean includeSqlTypes
+  ) throws IOException
   {
-    jsonGenerator.writeStartArray();
-
-    for (String columnName : columnNames) {
-      jsonGenerator.writeString(columnName);
-    }
-
-    jsonGenerator.writeEndArray();
+    ArrayWriter.writeHeader(jsonGenerator, rowType, includeTypes, includeSqlTypes);
   }
 
   @Override

sql/src/main/java/org/apache/druid/sql/http/ArrayWriter.java

@@ -21,11 +21,13 @@
 
 import com.fasterxml.jackson.core.JsonGenerator;
 import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.sql.calcite.table.RowSignatures;
 
 import javax.annotation.Nullable;
 import java.io.IOException;
 import java.io.OutputStream;
-import java.util.List;
 
 public class ArrayWriter implements ResultFormat.Writer
 {
@@ -58,15 +60,13 @@ public void writeResponseEnd() throws IOException
   }
 
   @Override
-  public void writeHeader(final List<String> columnNames) throws IOException
+  public void writeHeader(
+      final RelDataType rowType,
+      final boolean includeTypes,
+      final boolean includeSqlTypes
+  ) throws IOException
   {
-    jsonGenerator.writeStartArray();
-
-    for (String columnName : columnNames) {
-      jsonGenerator.writeString(columnName);
-    }
-
-    jsonGenerator.writeEndArray();
+    writeHeader(jsonGenerator, rowType, includeTypes, includeSqlTypes);
   }
 
   @Override
@@ -92,4 +92,36 @@ public void close() throws IOException
   {
     jsonGenerator.close();
   }
+
+  static void writeHeader(
+      final JsonGenerator jsonGenerator,
+      final RelDataType rowType,
+      final boolean includeTypes,
+      final boolean includeSqlTypes
+  ) throws IOException
+  {
+    final RowSignature signature = RowSignatures.fromRelDataType(rowType.getFieldNames(), rowType);
+
+    jsonGenerator.writeStartArray();
+    for (String columnName : signature.getColumnNames()) {
+      jsonGenerator.writeString(columnName);
+    }
+    jsonGenerator.writeEndArray();
+
+    if (includeTypes) {
+      jsonGenerator.writeStartArray();
+      for (int i = 0; i < signature.size(); i++) {
+        jsonGenerator.writeString(signature.getColumnType(i).get().asTypeString());
+      }
+      jsonGenerator.writeEndArray();
+    }
+
+    if (includeSqlTypes) {
+      jsonGenerator.writeStartArray();
+      for (int i = 0; i < signature.size(); i++) {
+        jsonGenerator.writeString(rowType.getFieldList().get(i).getType().getSqlTypeName().getName());
+      }
+      jsonGenerator.writeEndArray();
+    }
+  }
 }

sql/src/main/java/org/apache/druid/sql/http/CsvWriter.java

@@ -20,6 +20,9 @@
 package org.apache.druid.sql.http;
 
 import com.opencsv.CSVWriter;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.druid.segment.column.RowSignature;
+import org.apache.druid.sql.calcite.table.RowSignatures;
 
 import javax.annotation.Nullable;
 import java.io.BufferedWriter;
@@ -59,9 +62,35 @@ public void writeResponseEnd() throws IOException
   }
 
   @Override
-  public void writeHeader(final List<String> columnNames)
+  public void writeHeader(
+      final RelDataType rowType,
+      final boolean includeTypes,
+      final boolean includeSqlTypes
+  )
   {
-    writer.writeNext(columnNames.toArray(new String[0]), false);
+    final RowSignature signature = RowSignatures.fromRelDataType(rowType.getFieldNames(), rowType);
+
+    writer.writeNext(signature.getColumnNames().toArray(new String[0]), false);
+
+    if (includeTypes) {
+      final String[] types = new String[rowType.getFieldCount()];
+
+      for (int i = 0; i < signature.size(); i++) {
+        types[i] = signature.getColumnType(i).get().asTypeString();
+      }
+
+      writer.writeNext(types, false);
+    }
+
+    if (includeSqlTypes) {
+      final String[] sqlTypes = new String[rowType.getFieldCount()];
+
+      for (int i = 0; i < signature.size(); i++) {
+        sqlTypes[i] = rowType.getFieldList().get(i).getType().getSqlTypeName().getName();
+      }
+
+      writer.writeNext(sqlTypes, false);
+    }
   }
 
   @Override

sql/src/main/java/org/apache/druid/sql/http/ObjectLinesWriter.java

@@ -22,11 +22,11 @@
 import com.fasterxml.jackson.core.JsonGenerator;
 import com.fasterxml.jackson.core.io.SerializedString;
 import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.calcite.rel.type.RelDataType;
 
 import javax.annotation.Nullable;
 import java.io.IOException;
 import java.io.OutputStream;
-import java.util.List;
 
 public class ObjectLinesWriter implements ResultFormat.Writer
 {
@@ -57,15 +57,13 @@ public void writeResponseEnd() throws IOException
   }
 
   @Override
-  public void writeHeader(final List<String> columnNames) throws IOException
+  public void writeHeader(
+      final RelDataType rowType,
+      final boolean includeTypes,
+      final boolean includeSqlTypes
+  ) throws IOException
   {
-    jsonGenerator.writeStartObject();
-
-    for (String columnName : columnNames) {
-      jsonGenerator.writeNullField(columnName);
-    }
-
-    jsonGenerator.writeEndObject();
+    ObjectWriter.writeHeader(jsonGenerator, rowType, includeTypes, includeSqlTypes);
   }
 
   @Override