fix(ingest/redshift): patch instead of replace redshift custom properties #9293

ethan-cartwright · 2023-11-22T22:41:57Z

Testing

added tests
tested this locally to ensure it works for:
- custom properties
- that table descriptions and other set methods function as expected
- the patch_custom_properties flag works as expected

Checklist

The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
Links to related issues (if applicable)
Tests for the changes have been added/updated (if applicable)
Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

asikowitz

At a quick glance, seems like we're no longer setting some properties we should be setting

metadata-ingestion/src/datahub/ingestion/source/redshift/redshift.py

…e, created, lastModified

metadata-ingestion/src/datahub/ingestion/api/source_helpers.py

smoke-test/tests/patch/test_dataset_patches.py

metadata-ingestion/src/datahub/ingestion/source/redshift/redshift.py

metadata-ingestion/tests/unit/test_redshift_source.py

….com/ethan-cartwright/datahub into add_patch_custom_properties_redshift

metadata-ingestion/src/datahub/ingestion/api/source_helpers.py

hsheth2

lgtm once the two small tweaks are made, and assuming the patch changes have been tested against a live datahub system

hsheth2 · 2024-03-05T18:15:40Z

metadata-ingestion/src/datahub/ingestion/source/redshift/redshift.py

+                )
+        else:
+            if custom_properties:
+                dataset_properties.customProperties = custom_properties


I think this is redundant, since we set it above

hsheth2 · 2024-03-05T18:17:54Z

metadata-ingestion/tests/unit/test_redshift_source.py

+from datahub.ingestion.source.redshift.redshift_schema import RedshiftTable
+
+
+@pytest.fixture


this probably should be a helper method instead of a fixture

….com/ethan-cartwright/datahub into add_patch_custom_properties_redshift

Ethan Cartwright added 5 commits November 21, 2023 17:01

add patch for custom properties for redshift ingest

c7c8560

remove unnecessary imports

61ce0c4

remove unneeded dataset_properties mwu

02ba8d7

add unit test

4e19c6b

clean up imports

314fb2d

github-actions bot added the ingestion PR or Issue related to the ingestion of metadata label Nov 22, 2023

ethan-cartwright changed the title ~~Patch instead of replace redshift custom properties~~ fix:patch instead of replace redshift custom properties Nov 22, 2023

ethan-cartwright changed the title ~~fix:patch instead of replace redshift custom properties~~ fix: patch instead of replace redshift custom properties Nov 22, 2023

ethan-cartwright changed the title ~~fix: patch instead of replace redshift custom properties~~ fix(ingest/redshift): patch instead of replace redshift custom properties Nov 22, 2023

asikowitz reviewed Nov 22, 2023

View reviewed changes

metadata-ingestion/src/datahub/ingestion/source/redshift/redshift.py Outdated Show resolved Hide resolved

metadata-ingestion/src/datahub/ingestion/source/redshift/redshift.py Outdated Show resolved Hide resolved

vercel bot deployed to Preview November 22, 2023 23:01 View deployment

WIP: add back some fields, TODO: create set methods for qualified nam…

5f4ccee

…e, created, lastModified

vercel bot deployed to Preview November 27, 2023 15:53 View deployment

add patch methods

ec0be9d

vercel bot deployed to Preview December 7, 2023 19:55 View deployment

Ethan Cartwright added 2 commits December 12, 2023 12:29

tests passing

c796133

add tests

7074a29

vercel bot deployed to Preview December 12, 2023 17:49 View deployment

fix linting

a8e10fe

vercel bot deployed to Preview December 13, 2023 02:35 View deployment

Ethan Cartwright added 2 commits December 26, 2023 11:03

Merge branch 'master' into add_patch_custom_properties_redshift

0e98060

refactor to helper method

ce36600

vercel bot had a problem deploying to Preview December 26, 2023 18:08 Failure

Ethan Cartwright added 3 commits December 28, 2023 10:47

Merge branch 'master' into add_patch_custom_properties_redshift

7d2b3e7

generalize typing to avoid circular dependency

763789a

remove unnecessary logs and imports

5954997

vercel bot deployed to Preview December 28, 2023 19:32 View deployment

hsheth2 reviewed Jan 2, 2024

View reviewed changes

Merge branch 'master' into add_patch_custom_properties_redshift

8e583eb

vercel bot deployed to Preview January 25, 2024 15:27 View deployment

vercel bot deployed to Preview February 5, 2024 19:43 View deployment

anshbansal added 3 commits February 27, 2024 18:07

fix lint source

4c2a9c0

Merge branch 'master' into add_patch_custom_properties_redshift

6ee9fe9

some lint fix

e2728e3

vercel bot deployed to Preview February 27, 2024 13:00 View deployment

Ethan Cartwright added 2 commits February 27, 2024 19:48

Merge branch 'master' into add_patch_custom_properties_redshift

01c5ad7

Merge branch 'add_patch_custom_properties_redshift' of https://github…

d4868a3

….com/ethan-cartwright/datahub into add_patch_custom_properties_redshift

vercel bot deployed to Preview February 28, 2024 01:12 View deployment

reverted smoke tests

efaa28c

ethan-cartwright requested a review from hsheth2 February 28, 2024 06:13

vercel bot deployed to Preview February 28, 2024 06:37 View deployment

hsheth2 reviewed Feb 28, 2024

View reviewed changes

metadata-ingestion/src/datahub/ingestion/api/source_helpers.py Outdated Show resolved Hide resolved

Ethan Cartwright added 3 commits March 3, 2024 19:26

genericize dataset props patch builder

95ee34e

address linting

8a55ea7

remove log statement

738dd9a

ethan-cartwright requested a review from hsheth2 March 4, 2024 01:16

vercel bot deployed to Preview March 4, 2024 01:45 View deployment

black formatting

e942648

vercel bot deployed to Preview March 4, 2024 13:30 View deployment

Merge branch 'master' into add_patch_custom_properties_redshift

d7e64de

vercel bot deployed to Preview March 5, 2024 11:24 View deployment

enable patch by default

6a65d2a

vercel bot deployed to Preview March 5, 2024 11:44 View deployment

hsheth2 approved these changes Mar 5, 2024

View reviewed changes

Ethan Cartwright added 2 commits March 7, 2024 09:28

address PR comments

9723412

Merge branch 'add_patch_custom_properties_redshift' of https://github…

e396c56

….com/ethan-cartwright/datahub into add_patch_custom_properties_redshift

vercel bot deployed to Preview March 7, 2024 14:44 View deployment

address linter

89cf4c6

vercel bot deployed to Preview March 7, 2024 16:30 View deployment

hsheth2 merged commit 7540e64 into datahub-project:master Mar 11, 2024
53 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(ingest/redshift): patch instead of replace redshift custom properties #9293

fix(ingest/redshift): patch instead of replace redshift custom properties #9293

ethan-cartwright commented Nov 22, 2023 •

edited

asikowitz left a comment

hsheth2 left a comment

hsheth2 Mar 5, 2024

hsheth2 Mar 5, 2024

		from datahub.ingestion.source.redshift.redshift_schema import RedshiftTable


		@pytest.fixture

fix(ingest/redshift): patch instead of replace redshift custom properties #9293

fix(ingest/redshift): patch instead of replace redshift custom properties #9293

Conversation

ethan-cartwright commented Nov 22, 2023 • edited

Testing

Checklist

asikowitz left a comment

Choose a reason for hiding this comment

hsheth2 left a comment

Choose a reason for hiding this comment

hsheth2 Mar 5, 2024

Choose a reason for hiding this comment

hsheth2 Mar 5, 2024

Choose a reason for hiding this comment

ethan-cartwright commented Nov 22, 2023 •

edited