Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: warehouse_jobs support for async warehouse jobs #2319

Merged
merged 101 commits into from Oct 11, 2022

Conversation

chaitanyapraveen
Copy link
Contributor

@chaitanyapraveen chaitanyapraveen commented Aug 19, 2022

Description

Support for Asynchronous warehouse jobs to be triggered by sources(for now). The design document is attached here

Warehouse_jobs package

  1. Support for post,status, stop(incomplete) and get(incomplete)
  2. Support for various kinds of future warehouse jobs
  3. Modifications to pgNotifier to take tasks from warehouse_jobs runner
  4. Provision for slaves to selectively run warehouse_jobs based on the payload

The details on the tests are pending. Will update them shortly

Notion Ticket

https://www.notion.so/rudderstacks/Warehouse-Generic-De-dup-Framework-8d22fced31454568a6278679a43e97a1

Security

  • The code changed/added as part of this pull request won't create any security issues with how the software is being used.

@codecov
Copy link

codecov bot commented Aug 19, 2022

Codecov Report

Base: 45.30% // Head: 44.29% // Decreases project coverage by -1.00% ⚠️

Coverage data is based on head (f9ba0b2) compared to base (60bc1be).
Patch coverage: 26.02% of modified lines in pull request are covered.

❗ Current head f9ba0b2 differs from pull request most recent head 47e4d61. Consider uploading reports for the commit 47e4d61 to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2319      +/-   ##
==========================================
- Coverage   45.30%   44.29%   -1.01%     
==========================================
  Files         177      181       +4     
  Lines       36542    38350    +1808     
==========================================
+ Hits        16555    16987     +432     
- Misses      18911    20288    +1377     
+ Partials     1076     1075       -1     
Impacted Files Coverage Δ
gateway/gateway.go 54.24% <0.00%> (-0.53%) ⬇️
warehouse/admin.go 3.03% <0.00%> (ø)
warehouse/bigquery/bigquery.go 0.00% <0.00%> (ø)
warehouse/clickhouse/clickhouse.go 0.00% <0.00%> (ø)
warehouse/deltalake/deltalake.go 0.00% <0.00%> (ø)
warehouse/identities.go 1.06% <0.00%> (-0.46%) ⬇️
warehouse/jobs/handlers.go 0.00% <0.00%> (ø)
warehouse/jobs/jobs.go 0.00% <0.00%> (ø)
warehouse/jobs/runner.go 0.00% <0.00%> (ø)
warehouse/mssql/mssql.go 0.44% <0.00%> (-0.02%) ⬇️
... and 22 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

Copy link
Contributor

@abhimanyubabbar abhimanyubabbar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1st Chunk of overall PR review.

gateway/gateway.go Outdated Show resolved Hide resolved
gateway/gateway.go Outdated Show resolved Hide resolved
gateway/gateway.go Outdated Show resolved Hide resolved
gateway/gateway.go Outdated Show resolved Hide resolved
services/pgnotifier/pgnotifier.go Outdated Show resolved Hide resolved
warehouse/azure-synapse/azure-synapse.go Outdated Show resolved Hide resolved
warehouse/bigquery/bigquery.go Outdated Show resolved Hide resolved
warehouse/bigquery/bigquery.go Outdated Show resolved Hide resolved
warehouse/clickhouse/clickhouse.go Outdated Show resolved Hide resolved
warehouse/manager/manager.go Outdated Show resolved Hide resolved
warehouse/warehouse_jobs/runner.go Outdated Show resolved Hide resolved
warehouse/warehouse_jobs/runner.go Outdated Show resolved Hide resolved
warehouse/warehouse_jobs/runner.go Outdated Show resolved Hide resolved
warehouse/warehouse_jobs/runner.go Outdated Show resolved Hide resolved
@chaitanyapraveen chaitanyapraveen marked this pull request as ready for review September 7, 2022 10:57
@achettyiitr achettyiitr dismissed abhimanyubabbar’s stale review October 11, 2022 06:48

All changes are addressed. Seems unavailable at the moment.

@chaitanyapraveen chaitanyapraveen merged commit 73cc48a into master Oct 11, 2022
@github-actions github-actions bot deleted the feat.warehouse_jobs branch December 12, 2022 02:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants