SparkSql Editor fetch_result issues with complex types - Hue 4.10

[elisabetao]

Describe the bug:
When querying complex types in the latest Spark SQL Editor in Hue 4.10 the results are not being displayed by the _fetch_result method in desktop/libs/notebook/src/notebook/connectors/spark_shell.py

Steps to reproduce it?
Create a table with struct, array ort map types and query it though the SparkSQL Livy Editor

For now I am setting the type to string in case field['type']) is map or array, but maybe a better parser could be used for results fetching in SparkSQL Editor. Have anyone else encountered this issue?

Thanks a lot

[Harshg999]

Hi @elisabetao,
There has been some improvements recently for SparkSQL, can you once check with the recent master branch? Hue 4.11 is not out yet.

[elisabetao]

HI @Harshg999 Harsh,

Sorry for the delay, yes I have tested the latest code in the master branch in github , the issues is still there for complex types. Here is how it looks

You can test also on your side with an example as follows:
create table default.structeo(id int, name string, address structcity:string,State:string)
insert into table default.structeo select 1, 'Test', named_struct('city','Toronto','State','ON') from default.dummy_TBL limit 1
select * from default.structeo

Thanks a lot

[divincode]

Hi @elisabetao what is the expected output I will try to reproduce it and look into it.Can u also share your ini file configuration.

[elisabetao]

Hi @divincode , thanks for the update .

I would expect to have the same results as in Hive :

I think it's similar to the issue you remarked also for Trino/Presto.

On my side to at least allow the display of the data in desktop/libs/notebook/src/notebook/connectors/spark_shell.py I updated the following in _fetch_result, as a quick hack/workaround :

if 'struct' in str(field['type']):
              meta.append({'name': field['name'], 'type': 'struct' , 'comment': ''})
           elif 'array' in str(field['type']):
              meta.append({'name': field['name'], 'type': 'string' , 'comment': ''})
           elif 'map' in str(field['type']):
              meta.append({'name': field['name'], 'type': 'string' , 'comment': ''})
           else:
              meta.append({'name': field['name'], 'type': field['type'] , 'comment': ''})
          #meta = [{'name': field['name'], 'type': field['type'], 'comment': ''} for field in result['schema']['fields']]

but parsers need to be handled for all non Hive editors for Complex types.
Here's the part of hue.ini that is of interest:

[[interpreters]]
   # Define the name and how to connect and execute the language.
   # http://cloudera.github.io/hue/latest/administrator/configuration/editor/
     [[[sql]]]
      name=SparkSql
      interface=livy

[spark]
 # The Livy Server URL.
 livy_server_url=https://example.domain:8999/

 # Configure Livy to start in local 'process' mode, or 'yarn' workers.
 livy_server_session_kind=yarn

 # Whether Livy requires client to perform Kerberos authentication.
 security_enabled=true

 # Whether Livy requires client to use csrf protection.
 ## csrf_enabled=false

 # Host of the Sql Server
 sql_server_host=example.domain

 # Port of the Sql Server
 sql_server_port=10090

 livy_impersonation_enabled=true
 
 # Choose whether Hue should validate certificates received from the server.
 ssl_cert_ca_verify=false

 # Use SASL framework to establish connection to host.
 ## use_sasl=false

[github-actions]

This issue is stale because it has been open 30 days with no activity and is not "roadmap" labeled or part of any milestone. Remove stale label or comment or this will be closed in 5 days.

[Harshg999]

Hi @elisabetao, thank you for opening this issue!
Complex types query result support is now merged in master via #3097. It will also be available in upcoming 4.11 release.