DRILL-7604: Allow session options to be set in HTTP queries

[dobesv]

DRILL-7604: Allow session options to be set in HTTP queries

Description

Normally session options must be set using the SET command. However,
this command does not work for HTTP clients (REST API and Web UI)
because it doesn't keep a session between requests.

This changes it so that a request can contain the options you might want
to set using the SET command. It also updates the Web UI to include a
widget to select the store.format option so users can specify the file
format that will be used be CREATE TABLE AS. Additional options could
be added later if desired without much difficulty.

Documentation

POST /query.json

Submit a query and return results.

Parameters

• queryType--SQL, PHYSICAL, or LOGICAL are valid types. Use only "SQL". Other types are for internal use only.
• query--A SQL query that runs in Drill.
• autoLimit--Limits the number of rows returned from the result set. (Drill 1.16+)
• options--Object containing session options that you could have set using the SET command. Note that numeric & boolean
  options should be provided as a number or boolean, not as a string. A common iuse case would be to set "store.format":"json"
  to specify that CREATE TABLE AS should output in JSON format.

Testing

I added a JUnit test. I also did some manual testing.

(Please describe how this PR has been tested.)

[dobesv]

The test pass on my machine, does anyone have any hint on how I could try to reproduce the failure?

[paul-rogers]

Wonderful addition to the REST API. One suggestion about simplifying the code, another about the list of CREATE TABLE formats.

[arina-ielchiieva]

@dobesv also please notice that new tests fail on GitHub Actions, please make sure tests pass.

[dobesv]

I don't know how to fix the tests, I need some help with that.

[paul-rogers]

Thanks again for the improvement and the thorough tests. Just a few nits this time around.

[paul-rogers]

@dobesv, thanks for the PR. We can always make it better, but I'll be mindful of your time.

Oh, realized we still have that crash:

Errors: 
  TestQueryWrapper.testShowSchemas:106->run:75->runWithOptions:95 » NullPointer
 TestQueryWrapper.testSpecifyOutputFormatAsJson:131->runWithOption:101->runWithOptions:95 » NullPointer
TestQueryWrapper.testSpecifyOutputFormatAsParquet:115->runWithOption:101->runWithOptions:95 » NullPointer

I'll see if I can poke around and find the cause. Once we fix that, we'll be good to go.

[dobesv]

@paul-rogers Thanks Paul, I appreciate your help. I'm not experienced with the project, and I'm quite out of date with my Java skills, so hopefully this is something you could debug in minutes whereas it could take me many hours.

[arina-ielchiieva]

@dobesv how user will set session options through web ui?

[dobesv]

This PR doesn't include a way to set options via the Web UI any more, I guess that'll be something for a future PR.

If we come up with a list of options that are popular to set then we could add those in follow-up PRs. I had an example of such a field in this PR originally but it turned out be problematic. I don't have a good replacement example option in mind currently.

[arina-ielchiieva]

@dobesv what about having empty box where user can set options?

[dobesv]

@arina-ielchiieva It sounds a bit complicated, maybe this PR can proceed without it and we can look into adding a new UI for setting options in the future.

[arina-ielchiieva]

@dobesv the problem is that I don't really like parsing of the options introduced in this PR (its partially duplicates work done in Drill plus does parsing in slightly different way which will lead to discrepancies in future), there is much cleaner way to parse options, you need to pass option name, its kind and value. Please refer to my previous comment https://github.com/apache/drill/pull/1997/files#r391122758 so this PR needs re-working.

[dobesv]

OK, I'll just set this aside for now.

[paul-rogers]

@dobesv, this is a super useful feature. Perhaps I can tackle the Drill-specific bits that @arina-ielchiieva requested. How would you like do this? Should I issue a PR against your branch? Clone this branch and push a new PR?

[dobesv]

@paul-rogers Feel free to make a new branch from this one and carry on from there.