feat(warehouse): temp table support for postgres #2964

achettyiitr · 2023-02-09T16:09:16Z

Description

Temp table support

Temp staging tables support for POSTGRES. It will make a lot of our pipeline simpler by not adding extra handling for CrashRecoveries, Dropping, etc.
Added separate components for LoadTable, LoadUserTable, and Diagnostic.
Added support for observability around loading tables for POSTGRES.
Make dedup optional based on destinationIDs.

Additional changes

Added a separate component called loadfile.Downloader. This will download the load files for a specific table during an upload.
- number of workers configured
Deepsource recommendations
- use %q instead of "%s" for quoted strings
- Unnecessary block

Notion Ticket

https://www.notion.so/rudderstacks/Use-temp-tables-for-POSTGRES-286d6889c2e744d78f6297110054b551?pvs=4

Security

The code changed/added as part of this pull request won't create any security issues with how the software is being used.

codecov · 2023-02-09T16:35:35Z

Codecov Report

Patch coverage: 45.62% and project coverage change: +0.27 🎉

Comparison is base (1d9f63b) 53.43% compared to head (7f5611b) 53.71%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2964      +/-   ##
==========================================
+ Coverage   53.43%   53.71%   +0.27%     
==========================================
  Files         342      346       +4     
  Lines       52747    53614     +867     
==========================================
+ Hits        28185    28797     +612     
- Misses      22947    23181     +234     
- Partials     1615     1636      +21

Impacted Files	Coverage Δ
utils/misc/misc.go	`51.37% <ø> (-0.49%)`	⬇️
warehouse/identity/identity.go	`0.51% <0.00%> (+0.06%)`	⬆️
...ehouse/integrations/azure-synapse/azure-synapse.go	`0.31% <0.00%> (+0.02%)`	⬆️
warehouse/integrations/postgres/postgres.go	`0.53% <0.00%> (-28.10%)`	⬇️
warehouse/internal/service/recovery.go	`100.00% <ø> (ø)`
warehouse/upload.go	`23.58% <0.00%> (-0.02%)`	⬇️
warehouse/integrations/mssql/mssql.go	`1.62% <6.20%> (+0.66%)`	⬆️
warehouse/integrations/postgres-legacy/postgres.go	`28.62% <28.62%> (ø)`
warehouse/integrations/manager/manager.go	`51.19% <54.16%> (-3.10%)`	⬇️
...nternal/service/loadfiles/downloader/downloader.go	`82.55% <82.55%> (ø)`
... and 17 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

…re.postgres-temp-tables

achettyiitr · 2023-02-10T12:12:24Z

warehouse/integrations/azure-synapse/azure-synapse.go

@@ -135,7 +149,7 @@ func connect(cred credentialsT) (*sql.DB, error) {
 		Host:     net.JoinHostPort(cred.host, strconv.Itoa(port)),
 		RawQuery: query.Encode(),
 	}
-	pkgLogger.Debugf("synapse connection string : %s", connUrl.String())


This contains sensitive information like passwords and access keys. We should probably not log it.

achettyiitr · 2023-02-25T09:05:55Z

warehouse/integrations/mssql/mssql.go

@@ -151,7 +163,6 @@ func Connect(cred CredentialsT) (*sql.DB, error) {
 		Host:     net.JoinHostPort(cred.Host, strconv.Itoa(port)),
 		RawQuery: query.Encode(),
 	}
-	pkgLogger.Debugf("mssql connection string : %s", connUrl.String())


This contains sensitive information like Passwords, we should not probably log it.

lvrach · 2023-03-01T09:19:21Z

warehouse/internal/service/load_file_downloader/load_file_downloader.go

@@ -0,0 +1,132 @@
+package load_file_downloader


We can create a package for all things associated with loadfiles

Suggested change

package load_file_downloader

package loadfile

And then we can have Downloader inside this file.

lvrach · 2023-03-01T09:23:26Z

warehouse/internal/service/load_file_downloader/load_file_downloader.go

+		fileNamesLock sync.RWMutex
+	)
+
+	objects := l.uploader.GetLoadFilesMetadata(warehouseutils.GetLoadFilesOptionsT{Table: tableName})


Do we download all the files associated with an upload table ?

We download only the once corresponding to that table.

achettyiitr · 2023-03-01T13:28:23Z

warehouse/integrations/postgres/load.go

+
+	// Creating staging table
+	sortedColumnKeys = warehouseutils.SortColumnKeysFromColumnMap(tableSchemaInUpload)
+	stagingTableName = warehouseutils.StagingTableName(provider, tableName, tableNameLimit)


Flag for a temporary table.

We are now having legacy implementation back driven by config Warehouse.postgres.useLegacy set to true by default.

…re.postgres-temp-tables

added support for load file downloader

bf446e7

github-actions bot added warehouse-team with tests labels Feb 9, 2023

achettyiitr added 6 commits February 10, 2023 00:51

Merge branch 'master' of github.com:rudderlabs/rudder-server into cho…

b2c8909

…re.postgres-temp-tables

formatted mssql and azure synapse

ace9f82

added load table and load users table components

78141c0

some more changes

8e0499d

some more changes

223c416

added test cases for user and identifies table

1cbbcb2

achettyiitr marked this pull request as draft February 10, 2023 11:31

achettyiitr commented Feb 10, 2023

View reviewed changes

achettyiitr added 2 commits February 10, 2023 18:51

code cleanup

2170044

added some more tests.

fd6c055

achettyiitr requested review from abhimanyubabbar and lvrach February 10, 2023 14:35

achettyiitr added 5 commits February 15, 2023 12:48

master pull

6e7e693

some more test cases

e95967f

bq create table

9f0e8ca

handle in case of user table loading

88841a1

some more changes.

538ac83

achettyiitr requested a review from cisse21 February 25, 2023 08:43

achettyiitr added 3 commits February 25, 2023 14:15

added diagnostic

ccca5ca

make fmt

c7b6d09

master pull

8eac2f2

achettyiitr marked this pull request as ready for review February 25, 2023 09:02

achettyiitr commented Feb 25, 2023

View reviewed changes

achettyiitr added 2 commits February 25, 2023 14:41

make fmt

914647a

deepsource changes

af1398e

achettyiitr mentioned this pull request Feb 25, 2023

feat(warehouse): added additional datatypes for warehouse integrations #2971

Closed

1 task

achettyiitr added 3 commits February 26, 2023 22:52

added observability for dedup query

f89238b

review comments

bb38850

master pull

cc63f53

achettyiitr removed the request for review from abhimanyubabbar February 28, 2023 13:55

lvrach reviewed Mar 1, 2023

View reviewed changes

achettyiitr commented Mar 1, 2023

View reviewed changes

achettyiitr added 3 commits March 1, 2023 20:04

Merge branch 'master' of github.com:rudderlabs/rudder-server into cho…

483d3de

…re.postgres-temp-tables

move load files downloader insider the loadfiles package

193bc42

use legacy code back

fa45b5d

achettyiitr requested a review from lvrach March 1, 2023 14:58

imports issue fix

40d69cb

achettyiitr force-pushed the chore.postgres-temp-tables branch from e879999 to 40d69cb Compare March 1, 2023 15:19

achettyiitr added 7 commits March 1, 2023 21:25

Merge branch 'master' of github.com:rudderlabs/rudder-server into cho…

894db2c

…re.postgres-temp-tables

remove duplicated tests for postgres

9b9c7a0

master pull

aea5bca

added crash reccoveries changes back.

2e016c6

fix tests

dfeb215

master pull

e962846

fix test failures

ef25ae7

lvrach approved these changes Mar 7, 2023

View reviewed changes

achettyiitr added 2 commits March 7, 2023 21:01

Merge branch 'master' into chore.postgres-temp-tables

998a175

Merge branch 'master' into chore.postgres-temp-tables

7f5611b

cisse21 approved these changes Mar 9, 2023

View reviewed changes

achettyiitr merged commit 9a80f45 into master Mar 9, 2023

achettyiitr deleted the chore.postgres-temp-tables branch March 9, 2023 09:09

This was referenced Mar 14, 2023

chore: prerelease 1.7.0-preview.4 #3099

Merged

chore: release 1.7.0 #3106

Merged

chore: prerelease 1.7.0-rc.1 #3107

Merged

chore: prerelease 1.7.0-rc.2 #3114

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(warehouse): temp table support for postgres #2964

feat(warehouse): temp table support for postgres #2964

achettyiitr commented Feb 9, 2023 •

edited

codecov bot commented Feb 9, 2023 •

edited

achettyiitr Feb 10, 2023 •

edited

achettyiitr Feb 25, 2023

lvrach Mar 1, 2023

lvrach Mar 1, 2023

lvrach Mar 1, 2023

achettyiitr Mar 1, 2023 •

edited

achettyiitr Mar 1, 2023 •

edited

achettyiitr Mar 1, 2023

feat(warehouse): temp table support for postgres #2964

feat(warehouse): temp table support for postgres #2964

Conversation

achettyiitr commented Feb 9, 2023 • edited

Description

Temp table support

Additional changes

Notion Ticket

Security

codecov bot commented Feb 9, 2023 • edited

Codecov Report

achettyiitr Feb 10, 2023 • edited

Choose a reason for hiding this comment

achettyiitr Feb 25, 2023

Choose a reason for hiding this comment

lvrach Mar 1, 2023

Choose a reason for hiding this comment

lvrach Mar 1, 2023

Choose a reason for hiding this comment

lvrach Mar 1, 2023

Choose a reason for hiding this comment

achettyiitr Mar 1, 2023 • edited

Choose a reason for hiding this comment

achettyiitr Mar 1, 2023 • edited

Choose a reason for hiding this comment

achettyiitr Mar 1, 2023

Choose a reason for hiding this comment

achettyiitr commented Feb 9, 2023 •

edited

codecov bot commented Feb 9, 2023 •

edited

achettyiitr Feb 10, 2023 •

edited

achettyiitr Mar 1, 2023 •

edited

achettyiitr Mar 1, 2023 •

edited