Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: added trino and spark test #3525

Merged
merged 12 commits into from
Jul 13, 2023
Merged

chore: added trino and spark test #3525

merged 12 commits into from
Jul 13, 2023

Conversation

achettyiitr
Copy link
Member

@achettyiitr achettyiitr commented Jun 21, 2023

Description

  • Added Trino test for datalake
    • Test for the xitongsys version of the parquet-go is failing with the following error which is expected.
      io.trino.spi.TrinoException: Corrupted statistics for column \"context_destination_id\" in Parquet file
      
  • Added Spark test for datalake
    • Test for the xitongsys version of the parquet-go is passing as expected.

Notion Ticket

https://www.notion.so/rudderstacks/Investigate-parquet-issue-for-AWS-GLUE-Presto-trino-6c8280c81d3a47eb9d54138586ab44d0?pvs=4

Security

  • The code changed/added as part of this pull request won't create any security issues with how the software is being used.

@codecov
Copy link

codecov bot commented Jun 22, 2023

Codecov Report

Patch coverage has no change and project coverage change: +0.06 🎉

Comparison is base (ccb3ed2) 68.05% compared to head (c8d815b) 68.12%.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #3525      +/-   ##
==========================================
+ Coverage   68.05%   68.12%   +0.06%     
==========================================
  Files         318      318              
  Lines       50382    50382              
==========================================
+ Hits        34289    34324      +35     
+ Misses      13867    13835      -32     
+ Partials     2226     2223       -3     

see 9 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@achettyiitr achettyiitr force-pushed the chore.trino-test branch 2 times, most recently from 0df87f7 to 76b62a9 Compare June 22, 2023 23:00
@achettyiitr achettyiitr requested a review from lvrach June 22, 2023 23:16
@achettyiitr achettyiitr changed the title chore: added trino test chore: added trino and spark test Jun 23, 2023
require.Eventually(t, func() bool {
_, err := db.ExecContext(ctx, "SELECT 1")
return err == nil
}, 60*time.Second, 1*time.Second)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could try more often, we don't even have log pollution here. Every 100ms for example could work.


var count int64

require.NoError(t, testhelper.WithConstantRetries(func() error {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest using require.Eventually with a short interval here as well, for sure one that is a fraction of a second. Same applies for others below.

@achettyiitr achettyiitr merged commit cfee014 into master Jul 13, 2023
36 checks passed
@achettyiitr achettyiitr deleted the chore.trino-test branch July 13, 2023 10:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants