Support for Nested Schema of Lists and Structs

[vizkidd]

Hi! I am trying to store NCBI_BLAST data with arrow. In my use case the data is grouped and tabular with each group having duplicated data in ID columns (like name, ID, taxon etc). If apache-arrow could support nested lists and structs, I can create a schema which would eliminate such duplications. Pardon me if such a design is by choice

Component(s)

Archery, C++

[mapleFU]

Seems arrow has struct and list datatype, see: https://arrow.apache.org/docs/python/api/datatypes.html

[vizkidd]

Thank you. How can I make a List of Structs?

My Current schema looks like this :

 seq_info_type = arrow::struct_({
      arrow::field("num_alignments", arrow::int8()),
      arrow::field("seqids", arrow::struct_({arrow::field("qseqid", arrow::utf8()),
                                             arrow::field("sseqid", arrow::utf8())})),
      arrow::field("seqs", arrow::struct_({arrow::field("qseq", arrow::large_utf8()),
                                           arrow::field("sseq", arrow::large_utf8())})),
      arrow::field("strands", arrow::utf8()),
      arrow::field("lengths", arrow::struct_({arrow::field("qlen", arrow::int8()),
                                              arrow::field("slen", arrow::int8())})),
  });
  hsp_type = arrow::struct_({arrow::field("pident", arrow::float64()),
 /*fill with data*/
                             arrow::field("comp_adj_method", arrow::float64())});

  alignment_scores_type = arrow::list({hsp_type});

  blast_schema = arrow::schema({arrow::field("seq_info", seq_info_type),
                                arrow::field("hsps", hsp_type)});

I want to use alignment_scores_type in blast_schema instead of the struct hsp_type so that I can store some meta-info in seq_info_type and have a variable length List of Structs with the data. This was disallowed in 13.0.0 (yet to check in 14.0.1)