Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pubsub, Pulsar, Redis, Redshift, Rocket destinations : Enable DAT tests #12143

Merged
merged 20 commits into from
Apr 28, 2022
Merged

Pubsub, Pulsar, Redis, Redshift, Rocket destinations : Enable DAT tests #12143

merged 20 commits into from
Apr 28, 2022

Conversation

DoNotPanicUA
Copy link
Contributor

@DoNotPanicUA DoNotPanicUA commented Apr 19, 2022 •
edited

What

Enable DAT tests #9443

  • Pubsub destination
  • Pulsar destination
  • Redis destination
  • Redshift destination
  • Rocket destination

Recommended reading order

  1. x.java
  2. y.python

@github-actions github-actions bot added the area/connectors Connector related issues label Apr 19, 2022
@DoNotPanicUA

This comment was marked as outdated.

Sign in to view
@DoNotPanicUA

This comment was marked as outdated.

Sign in to view
@DoNotPanicUA

This comment was marked as outdated.

Sign in to view
@DoNotPanicUA

This comment was marked as outdated.

Sign in to view
@DoNotPanicUA

This comment was marked as outdated.

Sign in to view
@DoNotPanicUA DoNotPanicUA changed the title .... destinations : Enable DAT tests Pubsub, Pulsar, Redis, Redshift, Rocket destinations : Enable DAT tests Apr 20, 2022
@DoNotPanicUA

This comment was marked as outdated.

Sign in to view
@DoNotPanicUA DoNotPanicUA linked an issue Apr 20, 2022 that may be closed by this pull request
Run the new DAT(s) against all destinations and find destinations that are not compliant #9444
Closed
@DoNotPanicUA
Merge remote-tracking branch 'origin/master' into aleonets/dat-PR
50551d5 
# Conflicts:
#	airbyte-integrations/connectors/destination-redshift/src/test-integration/java/io/airbyte/integrations/destination/redshift/RedshiftCopyDestinationAcceptanceTest.java
@DoNotPanicUA

This comment was marked as outdated.

Sign in to view
@DoNotPanicUA

This comment was marked as outdated.

Sign in to view
@DoNotPanicUA

This comment was marked as outdated.

Sign in to view
@DoNotPanicUA

This comment was marked as outdated.

Sign in to view
@DoNotPanicUA

This comment was marked as outdated.

Sign in to view
@DoNotPanicUA

This comment was marked as outdated.

Sign in to view
@DoNotPanicUA
Copy link
Contributor Author

DoNotPanicUA commented Apr 28, 2022 •
edited by github-actions bot

/test connector=connectors/destination-rockset

🕑 connectors/destination-rockset https://github.com/airbytehq/airbyte/actions/runs/2239148697
✅ connectors/destination-rockset https://github.com/airbytehq/airbyte/actions/runs/2239148697
No Python unittests run

@DoNotPanicUA
Copy link
Contributor Author

DoNotPanicUA commented Apr 28, 2022 •
edited by github-actions bot

/test connector=connectors/destination-redis

🕑 connectors/destination-redis https://github.com/airbytehq/airbyte/actions/runs/2239149709
✅ connectors/destination-redis https://github.com/airbytehq/airbyte/actions/runs/2239149709
No Python unittests run

@DoNotPanicUA

This comment was marked as outdated.

Sign in to view
@DoNotPanicUA
Copy link
Contributor Author

DoNotPanicUA commented Apr 28, 2022 •
edited by github-actions bot

/test connector=connectors/destination-pubsub

🕑 connectors/destination-pubsub https://github.com/airbytehq/airbyte/actions/runs/2239151610
✅ connectors/destination-pubsub https://github.com/airbytehq/airbyte/actions/runs/2239151610
No Python unittests run

@DoNotPanicUA DoNotPanicUA marked this pull request as ready for review April 28, 2022 12:05
@DoNotPanicUA

This comment was marked as outdated.

Sign in to view
@DoNotPanicUA
Copy link
Contributor Author

DoNotPanicUA commented Apr 28, 2022 •
edited by github-actions bot

/test connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/2239700136
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/2239700136
Python tests coverage:

Name                                                                                                                            Stmts   Miss  Cover
---------------------------------------------------------------------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                                                                                          2      0   100%
normalization/transform_catalog/reserved_keywords.py                                                                               13      0   100%
normalization/transform_catalog/__init__.py                                                                                         2      0   100%
normalization/destination_type.py                                                                                                  13      0   100%
normalization/__init__.py                                                                                                           4      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/airbyte_protocol.py     124      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/__init__.py               1      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/__init__.py                      2      0   100%
normalization/transform_catalog/destination_name_transformer.py                                                                   155      8    95%
normalization/transform_config/transform.py                                                                                       159     31    81%
normalization/transform_catalog/table_name_registry.py                                                                            174     34    80%
normalization/transform_catalog/utils.py                                                                                           34      7    79%
normalization/transform_catalog/dbt_macro.py                                                                                       22      7    68%
normalization/transform_catalog/catalog_processor.py                                                                              147     80    46%
normalization/transform_catalog/transform.py                                                                                       61     38    38%
normalization/transform_catalog/stream_processor.py                                                                               534    345    35%
---------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                                            1447    550    62%

@DoNotPanicUA
Copy link
Contributor Author

DoNotPanicUA commented Apr 28, 2022 •
edited by github-actions bot

/test connector=connectors/destination-pulsar

🕑 connectors/destination-pulsar https://github.com/airbytehq/airbyte/actions/runs/2239702760
✅ connectors/destination-pulsar https://github.com/airbytehq/airbyte/actions/runs/2239702760
No Python unittests run

@DoNotPanicUA
Copy link
Contributor Author

Followup ticket for the Redshift tests bug #12434

@DoNotPanicUA DoNotPanicUA merged commit aab1533 into master Apr 28, 2022
@DoNotPanicUA DoNotPanicUA deleted the aleonets/dat-PR branch April 28, 2022 15:26
This was referenced Apr 28, 2022
Merged
Closed
suhomud pushed a commit that referenced this pull request May 23, 2022
@DoNotPanicUA @suhomud
Pubsub, Pulsar, Redis, Redshift, Rocket destinations : Enable DAT tes…
e4e11cc 
…ts (#12143)

* enable DAT tests for Pulsar

* Enable DAT test for pubsub, redis, redshift, rocket

* format

* fix normalized data fetch

* cover "other" result type for arrays

* remove deserialization because now we have already parsed node

* fix bugspot

* fix unicode case
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sashaNeshcheret sashaNeshcheret sashaNeshcheret approved these changes

@alexandr-shegeda alexandr-shegeda Awaiting requested review from alexandr-shegeda

Assignees
No one assigned
Labels
area/connectors Connector related issues
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Run the new DAT(s) against all destinations and find destinations that are not compliant
2 participants
@DoNotPanicUA @sashaNeshcheret