Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SAT: Replace validate_output_from_all_streams with empty_streams param #4897

Merged
merged 4 commits into from
Jul 22, 2021
Merged

SAT: Replace validate_output_from_all_streams with empty_streams param #4897

merged 4 commits into from
Jul 22, 2021

Conversation

keu
Copy link
Contributor

@keu keu commented Jul 21, 2021

What

Describe what the change is solving
It helps to add screenshots if it affects the frontend.

How

Describe the solution

Recommended reading order

  1. x.java
  2. y.python

Pre-merge Checklist

Expand the checklist which is relevant for this PR.

Connector checklist

  • Issue acceptance criteria met
  • PR name follows PR naming conventions
  • Secrets are annotated with airbyte_secret in the connector's spec
  • Credentials added to Github CI if needed and not already present. instructions for injecting secrets into CI.
  • Unit & integration tests added as appropriate (and are passing)
    • Community members: please provide proof of this succeeding locally e.g: screenshot or copy-paste acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • /test connector=connectors/<name> command as documented here is passing.
    • Community members can skip this, Airbyters will run this for you.
  • Code reviews completed
  • Documentation updated
    • README.md
    • docs/SUMMARY.md if it's a new connector
    • Created or updated reference docs in docs/integrations/<source or destination>/<name>.
    • Changelog in the appropriate page in docs/integrations/.... See changelog example
    • docs/integrations/README.md contains a reference to the new connector
    • Build status added to build page
  • Build is successful
  • Connector version bumped like described here
  • New Connector version released on Dockerhub by running the /publish command described here
  • No major blockers
  • PR merged into master branch
  • Follow up tickets have been created
  • Associated tickets have been closed & stakeholders notified

Connector Generator checklist

  • Issue acceptance criteria met
  • PR name follows PR naming conventions
  • If adding a new generator, add it to the list of scaffold modules being tested
  • The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
  • Documentation which references the generator is updated as needed.

@keu keu requested a review from sherifnada July 21, 2021 21:26
keu
keu commented Jul 21, 2021
View reviewed changes
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py Outdated

def _validate_expected_records(self, records, expected_records, flags):
"""
We expect
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably lost connection with my head somewhere here, will fix

sherifnada
sherifnada approved these changes Jul 21, 2021
View reviewed changes
Copy link
Contributor

@sherifnada sherifnada left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but can you add unit tests?

airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py
@@ -123,6 +123,52 @@ def primary_keys_for_records(streams, records):

@pytest.mark.default_timeout(5 * 60)
class TestBasicRead(BaseTest):
@staticmethod
def _validate_schema(records, configured_catalog):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unit tests?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unfortunately not for the whole test, but we have a test for validate_schema

@github-actions github-actions bot added the area/documentation Improvements or additions to documentation label Jul 22, 2021
@keu keu added Acceptance Tests https://docs.google.com/document/d/1pFLR8eLP5VOj6vDtWe69KZL0aWWHEA6Vkbod6NYJon4/edit# zazmic and removed area/documentation Improvements or additions to documentation labels Jul 22, 2021
@keu keu self-assigned this Jul 22, 2021
@keu
Copy link
Contributor Author

keu commented Jul 22, 2021 •
edited by github-actions bot

/publish connector=bases/source-acceptance-test

🕑 bases/source-acceptance-test https://github.com/airbytehq/airbyte/actions/runs/1057148384
✅ bases/source-acceptance-test https://github.com/airbytehq/airbyte/actions/runs/1057148384

@keu keu merged commit 0f51edb into master Jul 22, 2021
@keu keu deleted the keu/sat-explicit-empty-streams branch July 22, 2021 20:26
@sherifnada
Copy link
Contributor

@eugene-kulak I think this is breaking integration tests for other connectors, could you make sure we do a PR to remove the deleted property from their YAMLs?

@keu
Copy link
Contributor Author

keu commented Jul 22, 2021

@sherifnada on it ATM

@keu keu mentioned this pull request Jul 27, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sherifnada sherifnada sherifnada approved these changes

Assignees

@keu keu

Labels
Acceptance Tests https://docs.google.com/document/d/1pFLR8eLP5VOj6vDtWe69KZL0aWWHEA6Vkbod6NYJon4/edit#
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@keu @sherifnada @eugene-kulak