Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(warehouse): integration test changes v3 #2426

Merged
merged 14 commits into from Sep 19, 2022

Conversation

achettyiitr
Copy link
Member

Description

This contains series of changes related to Warehouse Integration Test:

  • Use warehouse-specific (wh_**) tables instead of gw_** or brt_** tables.
  • Checking for events in staging files, load files, table_uploads, and warehouse.
  • Removing gw and brt dependency.
  • Reduce Query costs as we are not directly Querying the warehouses.
  • Add configuration test as part of Integration tests.
  • Integration test for AZURE_DATALAKE using azurite emulator for spinning up the azure storage.
  • Integration test for GCS_DATALAKE using the cloud gcs for it.
  • Integration test for S3_DATALAKE using minio.

Notion Ticket

https://www.notion.so/rudderstacks/Integration-tests-v3-c9dcc33c577b4cb5806155429ff15378

Security

  • The code changed/added as part of this pull request won't create any security issues with how the software is being used.

@codecov
Copy link

codecov bot commented Sep 12, 2022

Codecov Report

Merging #2426 (762b8c5) into master (ac9afbd) will increase coverage by 0.22%.
The diff coverage is 42.85%.

@@            Coverage Diff             @@
##           master    #2426      +/-   ##
==========================================
+ Coverage   36.34%   36.56%   +0.22%     
==========================================
  Files         200      200              
  Lines       39843    39843              
==========================================
+ Hits        14479    14567      +88     
+ Misses      24454    24360      -94     
- Partials      910      916       +6     
Impacted Files Coverage Δ
warehouse/slave.go 1.25% <0.00%> (+0.83%) ⬆️
warehouse/upload.go 6.49% <0.00%> (ø)
warehouse/scheduling.go 73.33% <75.00%> (+16.29%) ⬆️
warehouse/constraint.go 89.23% <100.00%> (+89.23%) ⬆️
config/backend-config/namespace_config.go 76.33% <0.00%> (-2.30%) ⬇️
config/backend-config/backend-config.go 69.31% <0.00%> (-0.57%) ⬇️
enterprise/reporting/reporting.go 9.82% <0.00%> (+1.44%) ⬆️
enterprise/reporting/setup.go 52.38% <0.00%> (+14.28%) ⬆️
... and 1 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@achettyiitr achettyiitr changed the title chore(warehouse): integration test changes chore(warehouse): integration test changes v3 Sep 12, 2022
Copy link
Contributor

@abhimanyubabbar abhimanyubabbar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR includes major refactoring on the testcases, so I'll approve without much going in - depth.

"fmt"
"log"
"os"
"testing"

backendconfig "github.com/rudderlabs/rudder-server/config/backend-config"

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unnecessary space ?

@achettyiitr achettyiitr merged commit 4e98109 into master Sep 19, 2022
@github-actions github-actions bot deleted the chore.warehouseIntegrationtTestV3 branch November 17, 2022 02:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants