Regression in test_backward_compatibility

[monai]

Environment

• Airbyte version: 0.40.0-alpha
• OS Version / Instance: macOS
• Deployment: Docker
• Source Connector and version: source-amazon-seller-partner 0.2.25 (🎉 Source Amazon Seller Partner: add restock inventory reports #15063)
• Destination Connector and version: any
• Step where error happened: Acceptance tests

Current Behavior

Test test_backward_compatibility throws NonBackwardCompatibleError for newly added full_refresh_stream. It seems that this is a regression introduced in #15520.

Expected Behavior

The test should pass since the newly added stream is new, so it doesn't break any backward compatibility.

Logs

―――――――――――――――――――――――――――――――――――――――――― TestDiscovery.test_backward_compatibility[inputs0] ―――――――――――――――――――――――――――――――――――――――――――

self = <source_acceptance_test.tests.test_core.TestDiscovery object at 0x1056f01c0>, skip_backward_compatibility_tests = False
discovered_catalog = {'GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL': AirbyteStream(name='GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL', json...fresh'>], source_defined_cursor=None, default_cursor_field=None, source_defined_primary_key=None, namespace=None), ...}
previous_discovered_catalog = {'GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL': AirbyteStream(name='GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL', json...fresh'>], source_defined_cursor=None, default_cursor_field=None, source_defined_primary_key=None, namespace=None), ...}

    @pytest.mark.default_timeout(60)
    @pytest.mark.backward_compatibility
    def test_backward_compatibility(
        self,
        skip_backward_compatibility_tests: bool,
        discovered_catalog: MutableMapping[str, AirbyteStream],
        previous_discovered_catalog: MutableMapping[str, AirbyteStream],
    ):
        """Check if the current catalog is backward_compatible with the previous one."""
        assert isinstance(discovered_catalog, MutableMapping) and isinstance(previous_discovered_catalog, MutableMapping)
        checker = CatalogDiffChecker(previous_discovered_catalog, discovered_catalog)
>       checker.assert_is_backward_compatible()

../../bases/source-acceptance-test/source_acceptance_test/tests/test_core.py:321:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../bases/source-acceptance-test/source_acceptance_test/utils/backward_compatibility.py:217: in assert_is_backward_compatible
    self.check_if_cursor_field_was_changed(self.streams_cursor_fields_diff)
../../bases/source-acceptance-test/source_acceptance_test/utils/backward_compatibility.py:231: in check_if_cursor_field_was_changed
    self._raise_error("The value of 'default_cursor_field' was changed", diff)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <source_acceptance_test.utils.backward_compatibility.CatalogDiffChecker object at 0x10580c880>
message = "The value of 'default_cursor_field' was changed"
diff = {'dictionary_item_added': [<root['GET_RESTOCK_INVENTORY_RECOMMENDATIONS_REPORT'] t1:not present, t2:None>]}

    def _raise_error(self, message: str, diff: DeepDiff):
>       raise NonBackwardCompatibleError(f"{message}. Diff: {diff.pretty()}", self.context)
E       source_acceptance_test.utils.backward_compatibility.NonBackwardCompatibleError: BackwardIncompatibilityContext.DISCOVER - The value of 'default_cursor_field' was changed. Diff: Item root['GET_RESTOCK_INVENTORY_RECOMMENDATIONS_REPORT'] added to dictionary.

../../bases/source-acceptance-test/source_acceptance_test/utils/backward_compatibility.py:38: NonBackwardCompatibleError

 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_backward_compatibility[inputs0] ⨯92% █████████▎
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead.test_read[inputs0] ✓96% █████████▋{"type": "LOG", "log": {"level": "ERROR", "message": "Docker container was failed, code 1, error:\n{\"type\": \"TRACE\", \"trace\": {\"type\": \"ERROR\", \"emitted_at\": 1660657490591.721, \"error\": {\"message\": \"Something went wrong in the connector. See the logs for more details.\", \"internal_message\": \"2 validation errors for ConfiguredAirbyteCatalog\\nstreams -> 0 -> sync_mode\\n  value is not a valid enumeration member; permitted: 'full_refresh', 'incremental' (type=type_error.enum; enum_values=[<SyncMode.full_refresh: 'full_refresh'>, <SyncMode.incremental: 'incremental'>])\\nstreams -> 0 -> destination_sync_mode\\n  value is not a valid enumeration member; permitted: 'append', 'overwrite', 'append_dedup' (type=type_error.enum; enum_values=[<DestinationSyncMode.append: 'append'>, <DestinationSyncMode.overwrite: 'overwrite'>, <DestinationSyncMode.append_dedup: 'append_dedup'>])\", \"stack_trace\": \"Traceback (most recent call last):\\n  File \\\"/airbyte/integration_code/main.py\\\", line 13, in <module>\\n    launch(source, sys.argv[1:])\\n  File \\\"/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\\\", line 123, in launch\\n    for message in source_entrypoint.run(parsed_args):\\n  File \\\"/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\\\", line 111, in run\\n    config_catalog = self.source.read_catalog(parsed_args.catalog)\\n  File \\\"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/source.py\\\", line 54, in read_catalog\\n    return ConfiguredAirbyteCatalog.parse_obj(self.read_config(catalog_path))\\n  File \\\"/usr/local/lib/python3.9/site-packages/pydantic/main.py\\\", line 521, in parse_obj\\n    return cls(**obj)\\n  File \\\"/usr/local/lib/python3.9/site-packages/pydantic/main.py\\\", line 341, in __init__\\n    raise validation_error\\npydantic.error_wrappers.ValidationError: 2 validation errors for ConfiguredAirbyteCatalog\\nstreams -> 0 -> sync_mode\\n  value is not a valid enumeration member; permitted: 'full_refresh', 'incremental' (type=type_error.enum; enum_values=[<SyncMode.full_refresh: 'full_refresh'>, <SyncMode.incremental: 'incremental'>])\\nstreams -> 0 -> destination_sync_mode\\n  value is not a valid enumeration member; permitted: 'append', 'overwrite', 'append_dedup' (type=type_error.enum; enum_values=[<DestinationSyncMode.append: 'append'>, <DestinationSyncMode.overwrite: 'overwrite'>, <DestinationSyncMode.append_dedup: 'append_dedup'>])\\n\", \"failure_type\": \"system_error\"}}}\n"}}

 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead.test_airbyte_trace_message_on_failure[inputs0] ✓100% ██████████
{"type": "LOG", "log": {"level": "INFO", "message": "/Users/juozas/projects/airbyte/airbyte-integrations/connectors/source-amazon-seller-partner - SAT run - 68e552556a0ba92d024da69b05d1704699343e8d - FAILED"}}

======================================================== short test summary info ========================================================
SKIPPED [1] ../../bases/source-acceptance-test/source_acceptance_test/plugin.py:60: Skipping TestFullRefresh.test_sequential_reads because not found in the config
SKIPPED [1] ../../bases/source-acceptance-test/source_acceptance_test/plugin.py:60: Skipping TestIncremental.test_two_sequential_reads because not found in the config
FAILED ../../bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery::test_backward_compatibility[inputs0]

Results (74.00s):
      23 passed
       1 failed
         - airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py:310 TestDiscovery.test_backward_compatibility[inputs0]

Steps to Reproduce

1. Checkout 🎉 Source Amazon Seller Partner: add restock inventory reports #15063
2. Run docker build . -t airbyte/source-amazon-seller-partner:dev
3. Run python -m pytest -p integration_tests.acceptance

Are you willing to submit a PR?

No.

[marcosmarxm]

@alafanechere do you mind helping here?

[alafanechere]

@marcosmarxm and @monai I made the fix in #15835