DRILL-8204: Allow Provided Schema for HTTP Plugin in JSON Mode

[cgivre]

DRILL-8204: Allow Provided Schema for HTTP Plugin in JSON Mode

Description

See below. 👇

Documentation

Schema Provisioning

One of the challenges of querying APIs is inconsistent data. Drill allows you to provide a schema for individual endpoints. You can do this in one of three ways:

1. By providing a schema inline See: Specifying Schema as Table Function Parameter
2. By providing a schema in the configuration for the endpoint.

The schema provisioning currently supports complex types of Arrays and Maps at any nesting level.

Example Schema Provisioning:

"jsonOptions": {
  "schema": {
    "columns":[
      {
        "name":"outer_map",
        "type":"ARRAY<STRUCT<`bigint_col` BIGINT, `boolean_col` BOOLEAN, `date_col` DATE, `double_col` DOUBLE, `interval_col` INTERVAL, `int_col` BIGINT, `timestamp_col` TIMESTAMP, `time_col` TIME, `varchar_col` VARCHAR>>","mode":"REPEATED"
      }, {
        "name":"field1",
        "type":"VARCHAR",
        "mode":"OPTIONAL"
      },
    ]
  }
}

Dealing With Inconsistent Schemas

One of the major challenges of interacting with JSON data is when the schema is inconsistent. Drill has a UNION data type which is marked as experimental. At the time of
writing, the HTTP plugin does not support the UNION, however supplying a schema can solve a lot of those issues.

Json Mode

Drill offers the option of reading all JSON values as a string. While this can complicate downstream analytics, it can also be a more memory-efficient way of reading data with
inconsistent schema. Unfortunately, at the time of writing, JSON-mode is only available with a provided schema. However, future work will allow this mode to be enabled for
any JSON data.

Enabling JSON Mode:

You can enable JSON mode simply by adding the drill.json-mode property with a value of json to a field, as shown below:

"jsonOptions": {
  "readNumbersAsDouble": true,
  "schema": {
    "type": "tuple_schema",
      "columns": [
        {
          "name": "custom_fields",
          "type": "ARRAY<STRUCT<`value` VARCHAR PROPERTIES { 'drill.json-mode' = 'json' }>>",
          "mode": "REPEATED"
      }
    ]
  }
}

Testing

Added unit tests.

[vvysotskyi]

We should also support Specifying the Schema as Table Function Parameter and resolve the target schema if they both are specified, so some users could customize data types without accessing storage configs.

[jnturton]

This comment is similar to the last one from @vvysotskyi except that I was experimenting with the metastore. While I cannot see that CREATE SCHEMA could work for the http storage plugin, I think it is still possible in theory to store a schema in the metastore something like the following.

ANALYZE TABLE table(http.tpch_test_svc.`/nation.json` (type=>'json',
    schema=>'inline=(
        `n_nationkey`	INT not null,
        `n_name`	VARCHAR not null,
        `n_regionkey` 	DOUBLE not null,
        `n_comment` 	VARCHAR not null)'
    )) REFRESH METADATA;

Note that the above command works for local CSV but I got a Calcite error for a local JSON file. I did not test it with the HTTP plugin.

Some information about provided schema priority, should it be of interest: https://drill.apache.org/docs/using-drill-metastore/#schema-priority.

[vvysotskyi]

Metastore with JSON should also work fine, here is the unit test that checks it: TestMetastoreWithEasyFormatPlugin.testAnalyzeOnJsonTable.
For now, drill metastore supports only easy file formats and parquet, but in the future, it could handle the HTTP plugin.

[jnturton]

For now, drill metastore supports only easy file formats and parquet, but in the future, it could handle the HTTP plugin.

@vvysotskyi I think that the HTTP plugin makes use of the same readers as the easy format plugins (CSV, JSON, XML)? Does that mean that metastore might already work with HTTP, or are there likely to be pieces missing?

[cgivre]

@vvysotskyi Thanks for the review. I added the logic to the HttpBatchReader to support inline schema, however in doing so, I seem to have a found a bug with the SchemaNegotiator in that the schema always seems to be null with the inline schema.

I created DRILL-8205 to address this. I left the logic and a unit test so once we have a fix for DRILL-8205 it should all work as expected.

[cgivre]

@vvysotskyi Thanks for your comments and assistance! I got this to work and the HTTP plugin now supports inline schema!

[vvysotskyi]

Thanks for making changes, +1