Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] A struct column with nested document causes no data to return with connector ElasticSearch #1449

Open
brucedeen opened this issue Aug 30, 2023 · 0 comments
Assignees
Labels
bug Something isn't working

Comments

@brucedeen
Copy link

Describe the bug
I can add an Athena column (see #1448) to normalize my data structure, but there appears to be an odd limit of the number of attributes added and then no data will come back.

To Reproduce
Steps to reproduce the behavior:

  1. Create a complex struct under "Athena"
  2. Run a query SELECT athena.organization_id, athena
    FROM "lambda:dev-elasticsearch"."esdev"."invoices" as esInvoice
    LIMIT 1
  3. No Data returned

remove one column from the items struct and data will show up.
You add that column back in and remove one that came in and data will show up, indicating that it's not the columns but the number of columns that is the issue.

Expected behavior
That all columns would come back and the data associated.

Screenshots / Exceptions / Errors
Athena structure.

Connector Details (please complete the following information):

  • Version: 2023.30.1
  • Name elasticsearch connector.
  • Athena Query IDs

Additional context
The schema for the Athena column
[
{
"Name": "athena",
"Type": "struct<id:string,dob:string,organization_id:string,account_id:string,first_name:string,last_name:string,mobile_number:array<struct<phone_number:string,label:string>>,email:array<structemail:string,label:string>,oldest_unpaid_invoice:struct<due_date:string,invoice_id:string>,last_paper_statement:string,provider:struct<provider_id:bigint,name:string,account_id:string,phone_number:string,address:string,address2:string,state:string,city:string,postcode:string,stat:string,create_date:string,deleted_date:string,user_id:string,organization_id:string,type:string,remit_address2:string,remit_city:string,remit_name:string,remit_postcode:string,remit_state:string>,invoice:struct<invoice_id:string,due_date:string,paid_date:string,created_date:string,invoice_date:string,invoice_number:string,stat:string,amount:double,discount_amount:double,notes:string,integration_data:struct<original_billed_amount:double>,items:array<struct<item_id:string,quantity:int>>>>"
}
]

If I move id, dob as a sibling that will allow me to run queries and get data. If 1448 can be solved I think this issue will go completely away and reduce the amount of data I have to store in ElasticSearch.

I can also just remove id and dob and it will start having data show up.

@brucedeen brucedeen added the bug Something isn't working label Aug 30, 2023
@chngpe chngpe self-assigned this Mar 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants