🎉 New source: Fauna #15274

macmv · 2022-08-03T22:41:05Z

What

🚨 User Impact 🚨

This is a new source, so no user impact is expected.

Pre-merge Checklist

Tests

Unit

~/workplace/fauna-airbte/airbyte/airbyte-integrations/connectors/source-fauna (source-fauna) » python3.9 -m pytest                                                                                 cleve@Cleves-MBP
Test session starts (platform: darwin, Python 3.9.7, pytest 6.2.5, pytest-sugar 0.9.4)
cachedir: .pytest_cache
rootdir: /Users/cleve/workplace/fauna-airbte/airbyte, configfile: pytest.ini
plugins: sugar-0.9.4, timeout-1.4.2
collecting ... Expr({'now': None})
Expr({'paginate': Expr({'documents': Expr({'collection': 'foo'})}), 'size': 1})
Expr({'exists': Expr({'index': 'ts'})})
Expr({'select': 'source', 'from': Expr({'get': Expr({'index': 'ts'})})})
Expr({'select': 'values', 'from': Expr({'get': Expr({'index': 'ts'})}), 'default': Expr([])})
status=<Status.SUCCEEDED: 'SUCCEEDED'> message=None

 airbyte-integrations/connectors/source-fauna/unit_tests/check_test.py::test_valid_query ✓                                                                                                             4% ▌         
<MagicMock id='4486726992'>                                                                                                                                                                                        

 airbyte-integrations/connectors/source-fauna/unit_tests/check_test.py::test_invalid_check ✓                                                                                                           9% ▉         
Expr({'now': None})                                                                                                                                                                                                
Expr({'paginate': Expr({'documents': Expr({'collection': 'foo'})}), 'size': 1})
None
Expr({'now': None})
Expr({'paginate': Expr({'documents': Expr({'collection': 'invalid_collection_name'})}), 'size': 1})
Collection 'invalid_collection_name' does not exist
Expr({'now': None})
Expr({'paginate': Expr({'documents': Expr({'collection': 'foo'})}), 'size': 1})
Expr({'exists': Expr({'index': 'ts'})})
Expr({'select': 'source', 'from': Expr({'get': Expr({'index': 'ts'})})})
Expr({'select': 'values', 'from': Expr({'get': Expr({'index': 'ts'})}), 'default': Expr([])})
None
Expr({'now': None})
Expr({'paginate': Expr({'documents': Expr({'collection': 'foo'})}), 'size': 1})
Expr({'exists': Expr({'index': 'invalid_index_name'})})
Index 'invalid_index_name' does not exist
Expr({'now': None})
Expr({'paginate': Expr({'documents': Expr({'collection': 'foo'})}), 'size': 1})
Expr({'exists': Expr({'index': 'wrong_source'})})
An exception occurred: invalid query Expr({'exists': Expr({'index': 'wrong_source'})})
Expr({'now': None})
Expr({'paginate': Expr({'documents': Expr({'collection': 'foo'})}), 'size': 1})
Expr({'exists': Expr({'index': 'extra_values_index'})})
Expr({'select': 'source', 'from': Expr({'get': Expr({'index': 'extra_values_index'})})})
Expr({'select': 'values', 'from': Expr({'get': Expr({'index': 'extra_values_index'})}), 'default': Expr([])})
None
Expr({'now': None})
Expr({'paginate': Expr({'documents': Expr({'collection': 'foo'})}), 'size': 1})
Expr({'exists': Expr({'index': 'no_values_index'})})
Expr({'select': 'source', 'from': Expr({'get': Expr({'index': 'no_values_index'})})})
Expr({'select': 'values', 'from': Expr({'get': Expr({'index': 'no_values_index'})}), 'default': Expr([])})
Index should have values [{'field': 'ts'}, {'field': 'ref'}], but it has values []

 airbyte-integrations/connectors/source-fauna/unit_tests/check_test.py::test_check_fails ✓                                                                                                            13% █▍        
Expr({'now': None})                                                                                                                                                                                                
Expr({'paginate': Expr({'documents': Expr({'collection': 'foo'})}), 'size': 1})
Expr({'exists': Expr({'index': 'ts'})})
Expr({'select': 'source', 'from': Expr({'get': Expr({'index': 'ts'})})})
Expr({'select': 'values', 'from': Expr({'get': Expr({'index': 'ts'})}), 'default': Expr([])})
Expr({'now': None})
Expr({'paginate': Expr({'documents': Expr({'collection': 'foo'})}), 'size': 1})
Expr({'exists': Expr({'index': 'ts'})})
Expr({'select': 'source', 'from': Expr({'get': Expr({'index': 'ts'})})})
Expr({'select': 'values', 'from': Expr({'get': Expr({'index': 'ts'})}), 'default': Expr([])})

 airbyte-integrations/connectors/source-fauna/unit_tests/check_test.py::test_config_columns ✓                                                                                                         17% █▊        
Waiting for FaunaDB to start...                                                                                                                                                                                    
.............
FaunaDB is ready! Starting tests
Setting up database...
Database is setup!
[info]: reading add/removes for stream foo
[info]: emitting object {'ref': '105', 'ts': 1659566139180000, 'my_deletion_col': '2022-08-03T22:35:39.180000'}
Ref(id=339012264410481152, collection=Ref(id=foo, collection=Ref(id=collections))) Ref(id=339012264456618496, collection=Ref(id=foo, collection=Ref(id=collections))) Ref(id=339012264498561536, collection=Ref(id=foo, collection=Ref(id=collections)))
[info]: reading add/removes for stream foo
[info]: emitting object {'ref': '339012264410481152', 'ts': 1659566139650000, 'my_deletion_col': '2022-08-03T22:35:39.650000'}
[info]: emitting object {'ref': '339012264498561536', 'ts': 1659566139690000, 'my_deletion_col': '2022-08-03T22:35:39.690000'}
[info]: emitting object {'ref': '339012264456618496', 'ts': 1659566139730000, 'my_deletion_col': '2022-08-03T22:35:39.730000'}
=== check: make sure we read the initial state
[info]: state: {}
[info]: syncing stream 'foo' with incremental
[info]: skipping collection events (no deletions needed)
[info]: reading document updates for stream foo
=== check: make sure the state resumes
[info]: state: {'foo': {'updates_cursor': {'ts': 1659566138700000, 'ref': '104'}}}
[info]: syncing stream 'foo' with incremental
[info]: skipping collection events (no deletions needed)
[info]: reading document updates for stream foo
=== check: make sure that updates are actually read
[info]: state: {'foo': {'updates_cursor': {'ts': 1659566138700000, 'ref': '104'}}}
[info]: syncing stream 'foo' with incremental
[info]: skipping collection events (no deletions needed)
[info]: reading document updates for stream foo
=== check: make sure we read the initial state
[info]: state: {}
[info]: syncing stream 'deletions_test' with incremental
[info]: reading add/removes for stream deletions_test
[info]: reading document updates for stream deletions_test
=== check: make sure we don't produce more records when nothing changed
[info]: state: {'deletions_test': {'remove_cursor': {'ts': 0, 'ref': ''}, 'updates_cursor': {'ts': 1659566140490000, 'ref': '104'}}}
[info]: syncing stream 'deletions_test' with incremental
[info]: reading add/removes for stream deletions_test
[info]: reading document updates for stream deletions_test
=== check: make sure deleted documents produce records
[info]: state: {'deletions_test': {'remove_cursor': {'ts': 0, 'ref': ''}, 'updates_cursor': {'ts': 1659566140490000, 'ref': '104'}}}
[info]: syncing stream 'deletions_test' with incremental
[info]: reading add/removes for stream deletions_test
[info]: emitting object {'ref': '101', 'ts': 1659566140820000, 'deleted_at': '2022-08-03T22:35:40.820000'}
[info]: emitting object {'ref': '103', 'ts': 1659566140850000, 'deleted_at': '2022-08-03T22:35:40.850000'}
[info]: reading document updates for stream deletions_test
=== check: make sure we don't produce more deleted documents when nothing changed
[info]: state: {'deletions_test': {'remove_cursor': {'ts': 1659566140850000, 'ref': '103'}, 'updates_cursor': {'ts': 1659566140490000, 'ref': '104'}}}
[info]: syncing stream 'deletions_test' with incremental
[info]: reading add/removes for stream deletions_test
[info]: reading document updates for stream deletions_test
=== check: make sure another deleted document produces one more record
[info]: state: {'deletions_test': {'remove_cursor': {'ts': 1659566140850000, 'ref': '103'}, 'updates_cursor': {'ts': 1659566140490000, 'ref': '104'}}}
[info]: syncing stream 'deletions_test' with incremental
[info]: reading add/removes for stream deletions_test
[info]: emitting object {'ref': '102', 'ts': 1659566141020000, 'deleted_at': '2022-08-03T22:35:41.020000'}
[info]: reading document updates for stream deletions_test
=== check: make sure we don't produce more deleted documents when nothing changed
[info]: state: {'deletions_test': {'remove_cursor': {'ts': 1659566141020000, 'ref': '102'}, 'updates_cursor': {'ts': 1659566140490000, 'ref': '104'}}}
[info]: syncing stream 'deletions_test' with incremental
[info]: reading add/removes for stream deletions_test
[info]: reading document updates for stream deletions_test
Stopping FaunaDB container...
Stopped FaunaDB container

 airbyte-integrations/connectors/source-fauna/unit_tests/database_test.py::test_incremental_reads ✓                                                                                                   22% ██▎       
 airbyte-integrations/connectors/source-fauna/unit_tests/discover_test.py::test_simple_discover ✓                                                                                                     26% ██▋       
 airbyte-integrations/connectors/source-fauna/unit_tests/discover_test.py::test_discover_valid_index ✓                                                                                                30% ███▏      
 airbyte-integrations/connectors/source-fauna/unit_tests/discover_test.py::test_config_columns ✓                                                                                                      35% ███▌      
 airbyte-integrations/connectors/source-fauna/unit_tests/discover_test.py::test_discover_extra_columns ✓                                                                                              39% ███▉      
[info]: no after token, starting from beginning                                                                                                                                                                    

 airbyte-integrations/connectors/source-fauna/unit_tests/full_refresh_test.py::test_read_all ✓                                                                                                        43% ████▍     
[info]: no after token, starting from beginning                                                                                                                                                                    

 airbyte-integrations/connectors/source-fauna/unit_tests/full_refresh_test.py::test_read_all_extra_columns ✓                                                                                          48% ████▊     
[info]: no after token, starting from beginning                                                                                                                                                                    
[info]: using after token ['even more magical', 3, 'data']

 airbyte-integrations/connectors/source-fauna/unit_tests/full_refresh_test.py::test_read_all_resume ✓                                                                                                 52% █████▎    
[info]: state: {}                                                                                                                                                                                                  
[info]: syncing stream 'my_stream_name' with incremental

 airbyte-integrations/connectors/source-fauna/unit_tests/incremental_test.py::test_read_no_updates_or_creates_but_removes_present ✓                                                                   57% █████▋    
[info]: state: {}                                                                                                                                                                                                  
[info]: syncing stream 'my_stream_name' with incremental
[info]: skipping collection events (no deletions needed)

 airbyte-integrations/connectors/source-fauna/unit_tests/incremental_test.py::test_read_updates_ignore_deletes ✓                                                                                      61% ██████▏   
[info]: reading add/removes for stream foo                                                                                                                                                                         
[info]: emitting object {'ref': '100', 'ts': 99, 'deletes_here': '1970-01-01T00:00:00.000099'}
[info]: emitting object {'ref': '5', 'ts': 999, 'deletes_here': '1970-01-01T00:00:00.000999'}
[info]: reading add/removes for stream foo
[info]: emitting object {'ref': '3', 'ts': 12345, 'deletes_here': '1970-01-01T00:00:00.012345'}
[info]: reading add/removes for stream foo

 airbyte-integrations/connectors/source-fauna/unit_tests/incremental_test.py::test_read_removes_resume_from_partial_failure ✓                                                                         65% ██████▌   
[info]: reading add/removes for stream foo                                                                                                                                                                         
[info]: emitting object {'ref': '100', 'ts': 1648944000000000.0, 'my_deleted_column': '2022-04-03T00:00:00'}
[info]: reading add/removes for stream foo
[info]: reading add/removes for stream foo
[info]: emitting object {'ref': '300', 'ts': 1648944001000000.0, 'my_deleted_column': '2022-04-03T00:00:01'}

 airbyte-integrations/connectors/source-fauna/unit_tests/incremental_test.py::test_read_remove_deletions ✓                                                                                            70% ██████▉   
[info]: reading document updates for stream my_stream_name                                                                                                                                                         
[info]: reading document updates for stream my_stream_name
[info]: reading document updates for stream my_stream_name

 airbyte-integrations/connectors/source-fauna/unit_tests/incremental_test.py::test_read_updates_query ✓                                                                                               74% ███████▍  
[info]: reading document updates for stream my_stream_name                                                                                                                                                         
[info]: reading document updates for stream my_stream_name

 airbyte-integrations/connectors/source-fauna/unit_tests/incremental_test.py::test_read_updates_resume ✓                                                                                              78% ███████▉  
 airbyte-integrations/connectors/source-fauna/unit_tests/serialize_test.py::test_date ✓                                                                                                               83% ████████▍ 
 airbyte-integrations/connectors/source-fauna/unit_tests/serialize_test.py::test_fauna_time ✓                                                                                                         87% ████████▊ 
 airbyte-integrations/connectors/source-fauna/unit_tests/serialize_test.py::test_bytes ✓                                                                                                              91% █████████▎
 airbyte-integrations/connectors/source-fauna/unit_tests/serialize_test.py::test_ref ✓                                                                                                                96% █████████▋
 airbyte-integrations/connectors/source-fauna/unit_tests/serialize_test.py::test_recursive ✓                                                                                                         100% ██████████
================================================================================================= warnings summary =================================================================================================
../../../../../../.asdf/installs/python/3.9.7/lib/python3.9/site-packages/hyper/http11/connection.py:13
../../../../../../.asdf/installs/python/3.9.7/lib/python3.9/site-packages/hyper/http11/connection.py:13
  /Users/cleve/.asdf/installs/python/3.9.7/lib/python3.9/site-packages/hyper/http11/connection.py:13: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working
    from collections import Iterable, Mapping

../../../../../../.asdf/installs/python/3.9.7/lib/python3.9/site-packages/hyperframe/flags.py:14
  /Users/cleve/.asdf/installs/python/3.9.7/lib/python3.9/site-packages/hyperframe/flags.py:14: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working
    class Flags(collections.MutableSet):

../../../../../../.asdf/installs/python/3.9.7/lib/python3.9/site-packages/hyper/common/headers.py:13
  /Users/cleve/.asdf/installs/python/3.9.7/lib/python3.9/site-packages/hyper/common/headers.py:13: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working
    class HTTPHeaderMap(collections.MutableMapping):

-- Docs: https://docs.pytest.org/en/stable/warnings.html

Results (21.32s):
      23 passed

Integration

unit_tests/database_test.py tests exporting data from a local Fauna container. These tests pass, and are grouped in with the unit tests above.

Acceptance

~/workplace/fauna-airbte/airbyte/airbyte-integrations/connectors/source-fauna (source-fauna) » python3.9 -m pytest -p integration_tests.acceptance
Test session starts (platform: darwin, Python 3.9.7, pytest 6.2.5, pytest-sugar 0.9.4)
cachedir: .pytest_cache
rootdir: /Users/cleve/workplace/fauna-airbte/airbyte, configfile: pytest.ini
plugins: sugar-0.9.4, timeout-1.4.2
collecting ... 
 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_config_match_spec[inputs0] ✓3% ▍                                                                                                
 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_match_expected[inputs0] ✓6% ▋                                                                                                   
 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_docker_env[inputs0] ✓8% ▉                                                                                                       
 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_oneof_usage[inputs0] ✓11% █▎                                                                                                    
 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_required[inputs0] ✓14% █▍                                                                                                       
 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_optional[inputs0] ✓17% █▋                                                                                                       
 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_has_secret[inputs0] ✓19% █▉                                                                                                     
 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_secret_never_in_the_output[inputs0] ✓22% ██▎                                                                                    
 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_defined_refs_exist_in_json_spec_file[inputs0] ✓25% ██▌                                                                          
 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_oauth_flow_parameters[inputs0] ✓28% ██▊                                                                                         
 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestConnection.test_check[inputs0] ✓31% ███▏                                                                                                  
 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestConnection.test_check[inputs1] ✓33% ███▍                                                                                                  
 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestConnection.test_check[inputs2] ✓36% ███▋                                                         ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_discover[inputs0] ✓                                                39% ███▉      
 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_discover[inputs1] ✓                                       42% ████▎     
 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_cursors_exist_in_schema[inputs0] ✓                44% ████▌     
 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_cursors_exist_in_schema[inputs1] ✓                47% ████▊     
 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_refs_exist_in_schema[inputs0] ✓                   50% █████     
 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_refs_exist_in_schema[inputs1] ✓                   53% █████▍    
 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_keyword_exist_in_schema[inputs0-allOf] ✓          56% █████▋    
 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_keyword_exist_in_schema[inputs0-not] ✓            58% █████▉    
 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_keyword_exist_in_schema[inputs1-allOf] ✓          61% ██████▎   
 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_keyword_exist_in_schema[inputs1-not] ✓            64% ██████▍   
 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_primary_keys_exist_in_schema[inputs0] ✓                   67% ██████▋   
 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_primary_keys_exist_in_schema[inputs1] ✓                   69% ██████▉   
{"type": "LOG", "log": {"level": "INFO", "message": "Actual records for stream sample-data:"}}                                                                                                                     
{"type": "LOG", "log": {"level": "INFO", "message": "[\n {\n  \"ref\": \"337567897171787849\",\n  \"ts\": 1658188683585000,\n  \"data\": {\n   \"a\": 5,\n   \"nested\": {\n    \"value\": 15\n   }\n  },\n  \"my_a_col\": 5,\n  \"nested_column\": 15,\n  \"optional_name\": null\n },\n {\n  \"ref\": \"337567897172836425\",\n  \"ts\": 1658271973660000,\n  \"data\": {\n   \"a\": 6,\n   \"nested\": {\n    \"value\": 20\n   },\n   \"name\": \"hello world\"\n  },\n  \"my_a_col\": 6,\n  \"nested_column\": 20,\n  \"optional_name\": \"hello world\"\n },\n {\n  \"ref\": \"337567897172837449\",\n  \"ts\": 1658188683585000,\n  \"data\": {\n   \"a\": 7,\n   \"nested\": {\n    \"value\": 25\n   }\n  },\n  \"my_a_col\": 7,\n  \"nested_column\": 25,\n  \"optional_name\": null\n },\n {\n  \"ref\": \"337567897173885001\",\n  \"ts\": 1658188683585000,\n  \"data\": {\n   \"a\": 8,\n   \"nested\": {\n    \"value\": 30\n   }\n  },\n  \"my_a_col\": 8,\n  \"nested_column\": 30,\n  \"optional_name\": null\n }\n]"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Expected records for stream sample-data:"}}
{"type": "LOG", "log": {"level": "INFO", "message": "[\n {\n  \"ref\": \"337567897171787849\",\n  \"ts\": 1658188683585000,\n  \"data\": {\n   \"a\": 5,\n   \"nested\": {\n    \"value\": 15\n   }\n  },\n  \"my_a_col\": 5,\n  \"nested_column\": 15,\n  \"optional_name\": null\n },\n {\n  \"ref\": \"337567897172836425\",\n  \"ts\": 1658271973660000,\n  \"data\": {\n   \"a\": 6,\n   \"nested\": {\n    \"value\": 20\n   },\n   \"name\": \"hello world\"\n  },\n  \"my_a_col\": 6,\n  \"nested_column\": 20,\n  \"optional_name\": \"hello world\"\n },\n {\n  \"ref\": \"337567897172837449\",\n  \"ts\": 1658188683585000,\n  \"data\": {\n   \"a\": 7,\n   \"nested\": {\n    \"value\": 25\n   }\n  },\n  \"my_a_col\": 7,\n  \"nested_column\": 25,\n  \"optional_name\": null\n },\n {\n  \"ref\": \"337567897173885001\",\n  \"ts\": 1658188683585000,\n  \"data\": {\n   \"a\": 8,\n   \"nested\": {\n    \"value\": 30\n   }\n  },\n  \"my_a_col\": 8,\n  \"nested_column\": 30,\n  \"optional_name\": null\n }\n]"}}

 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead.test_read[inputs0] ✓                                           72% ███████▎  
{"type": "LOG", "log": {"level": "INFO", "message": "Actual records for stream deletions-data:"}}                                                                                                                  
{"type": "LOG", "log": {"level": "INFO", "message": "[\n {\n  \"ref\": \"338836293305763911\",\n  \"ts\": 1659398359360000,\n  \"deleted_at\": \"2022-08-01T23:59:19.360000\"\n },\n {\n  \"ref\": \"338836293305761863\",\n  \"ts\": 1659398366330000,\n  \"deleted_at\": \"2022-08-01T23:59:26.330000\"\n },\n {\n  \"ref\": \"338836293305765959\",\n  \"ts\": 1659398371330000,\n  \"deleted_at\": \"2022-08-01T23:59:31.330000\"\n },\n {\n  \"ref\": \"338836293305762887\",\n  \"ts\": 1659398320430000,\n  \"data\": {\n   \"a\": 6,\n   \"nested\": {\n    \"value\": 20\n   }\n  }\n },\n {\n  \"ref\": \"338836293305764935\",\n  \"ts\": 1659398320430000,\n  \"data\": {\n   \"a\": 8,\n   \"nested\": {\n    \"value\": 30\n   }\n  }\n }\n]"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Expected records for stream deletions-data:"}}
{"type": "LOG", "log": {"level": "INFO", "message": "[\n {\n  \"ref\": \"338836293305763911\",\n  \"ts\": 1659398359360000,\n  \"deleted_at\": \"2022-08-01T23:59:19.360000\"\n },\n {\n  \"ref\": \"338836293305761863\",\n  \"ts\": 1659398366330000,\n  \"deleted_at\": \"2022-08-01T23:59:26.330000\"\n },\n {\n  \"ref\": \"338836293305765959\",\n  \"ts\": 1659398371330000,\n  \"deleted_at\": \"2022-08-01T23:59:31.330000\"\n },\n {\n  \"ref\": \"338836293305762887\",\n  \"ts\": 1659398320430000,\n  \"data\": {\n   \"a\": 6,\n   \"nested\": {\n    \"value\": 20\n   }\n  }\n },\n {\n  \"ref\": \"338836293305764935\",\n  \"ts\": 1659398320430000,\n  \"data\": {\n   \"a\": 8,\n   \"nested\": {\n    \"value\": 30\n   }\n  }\n }\n]"}}

 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead.test_read[inputs1] ✓                                           75% ███████▌  
{"type": "LOG", "log": {"level": "ERROR", "message": "Docker container was failed, code 1, error:\n{\"type\": \"TRACE\", \"trace\": {\"type\": \"ERROR\", \"emitted_at\": 1659565939374.643, \"error\": {\"message\": \"Something went wrong in the connector. See the logs for more details.\", \"internal_message\": \"2 validation errors for ConfiguredAirbyteCatalog\\nstreams -> 0 -> sync_mode\\n  value is not a valid enumeration member; permitted: 'full_refresh', 'incremental' (type=type_error.enum; enum_values=[<SyncMode.full_refresh: 'full_refresh'>, <SyncMode.incremental: 'incremental'>])\\nstreams -> 0 -> destination_sync_mode\\n  value is not a valid enumeration member; permitted: 'append', 'overwrite', 'append_dedup' (type=type_error.enum; enum_values=[<DestinationSyncMode.append: 'append'>, <DestinationSyncMode.overwrite: 'overwrite'>, <DestinationSyncMode.append_dedup: 'append_dedup'>])\", \"stack_trace\": \"Traceback (most recent call last):\\n  File \\\"/airbyte/integration_code/main.py\\\", line 13, in <module>\\n    launch(source, sys.argv[1:])\\n  File \\\"/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\\\", line 118, in launch\\n    for message in source_entrypoint.run(parsed_args):\\n  File \\\"/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\\\", line 106, in run\\n    config_catalog = self.source.read_catalog(parsed_args.catalog)\\n  File \\\"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/source.py\\\", line 28, in read_catalog\\n    return ConfiguredAirbyteCatalog.parse_obj(self.read_config(catalog_path))\\n  File \\\"pydantic/main.py\\\", line 521, in pydantic.main.BaseModel.parse_obj\\n  File \\\"pydantic/main.py\\\", line 341, in pydantic.main.BaseModel.__init__\\npydantic.error_wrappers.ValidationError: 2 validation errors for ConfiguredAirbyteCatalog\\nstreams -> 0 -> sync_mode\\n  value is not a valid enumeration member; permitted: 'full_refresh', 'incremental' (type=type_error.enum; enum_values=[<SyncMode.full_refresh: 'full_refresh'>, <SyncMode.incremental: 'incremental'>])\\nstreams -> 0 -> destination_sync_mode\\n  value is not a valid enumeration member; permitted: 'append', 'overwrite', 'append_dedup' (type=type_error.enum; enum_values=[<DestinationSyncMode.append: 'append'>, <DestinationSyncMode.overwrite: 'overwrite'>, <DestinationSyncMode.append_dedup: 'append_dedup'>])\\n\", \"failure_type\": \"system_error\"}}}\n"}}

 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead.test_airbyte_trace_message_on_failure[inputs0] ✓               78% ███████▊  
{"type": "LOG", "log": {"level": "ERROR", "message": "Docker container was failed, code 1, error:\n{\"type\": \"TRACE\", \"trace\": {\"type\": \"ERROR\", \"emitted_at\": 1659565941167.9648, \"error\": {\"message\": \"Something went wrong in the connector. See the logs for more details.\", \"internal_message\": \"2 validation errors for ConfiguredAirbyteCatalog\\nstreams -> 0 -> sync_mode\\n  value is not a valid enumeration member; permitted: 'full_refresh', 'incremental' (type=type_error.enum; enum_values=[<SyncMode.full_refresh: 'full_refresh'>, <SyncMode.incremental: 'incremental'>])\\nstreams -> 0 -> destination_sync_mode\\n  value is not a valid enumeration member; permitted: 'append', 'overwrite', 'append_dedup' (type=type_error.enum; enum_values=[<DestinationSyncMode.append: 'append'>, <DestinationSyncMode.overwrite: 'overwrite'>, <DestinationSyncMode.append_dedup: 'append_dedup'>])\", \"stack_trace\": \"Traceback (most recent call last):\\n  File \\\"/airbyte/integration_code/main.py\\\", line 13, in <module>\\n    launch(source, sys.argv[1:])\\n  File \\\"/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\\\", line 118, in launch\\n    for message in source_entrypoint.run(parsed_args):\\n  File \\\"/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\\\", line 106, in run\\n    config_catalog = self.source.read_catalog(parsed_args.catalog)\\n  File \\\"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/source.py\\\", line 28, in read_catalog\\n    return ConfiguredAirbyteCatalog.parse_obj(self.read_config(catalog_path))\\n  File \\\"pydantic/main.py\\\", line 521, in pydantic.main.BaseModel.parse_obj\\n  File \\\"pydantic/main.py\\\", line 341, in pydantic.main.BaseModel.__init__\\npydantic.error_wrappers.ValidationError: 2 validation errors for ConfiguredAirbyteCatalog\\nstreams -> 0 -> sync_mode\\n  value is not a valid enumeration member; permitted: 'full_refresh', 'incremental' (type=type_error.enum; enum_values=[<SyncMode.full_refresh: 'full_refresh'>, <SyncMode.incremental: 'incremental'>])\\nstreams -> 0 -> destination_sync_mode\\n  value is not a valid enumeration member; permitted: 'append', 'overwrite', 'append_dedup' (type=type_error.enum; enum_values=[<DestinationSyncMode.append: 'append'>, <DestinationSyncMode.overwrite: 'overwrite'>, <DestinationSyncMode.append_dedup: 'append_dedup'>])\\n\", \"failure_type\": \"system_error\"}}}\n"}}

 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead.test_airbyte_trace_message_on_failure[inputs1] ✓               81% ████████▏ 
 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_full_refresh.py::TestFullRefresh.test_sequential_reads[inputs0] ✓                     83% ████████▍ 
 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental.test_two_sequential_reads[inputs0] ✓                  86% ████████▋ 
 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental.test_two_sequential_reads[inputs1] ✓                  89% ████████▉ 
 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental.test_read_sequential_slices[inputs0] ✓                92% █████████▎
 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental.test_read_sequential_slices[inputs1] ✓                94% █████████▌
 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental.test_state_with_abnormally_large_values[inputs0] ✓    97% █████████▊
 ../../fauna-airbyte-source/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental.test_state_with_abnormally_large_values[inputs1] ✓   100% ██████████
{"type": "LOG", "log": {"level": "INFO", "message": "/Users/cleve/workplace/fauna-airbte/airbyte/airbyte-integrations/connectors/source-fauna - SAT run - d9f845ec899f964faeba4d7791b59d05ec98f736 - PASSED"}}


Results (49.18s):
      36 passed

CLAassistant · 2022-08-03T22:41:10Z

All committers have signed the CLA.

marcosmarxm · 2022-08-15T23:22:21Z

@macmv sorry the long delay to review the contribution!

airbyte-integrations/connectors/source-fauna/source_fauna/spec.yaml

grishick

I added a couple of comments inline.

Additionally, please fix the issues reported by

./gradlew :airbyte-integrations:connectors:source-fauna:flakeCheck

Otherwise build task fails:

./gradlew :airbyte-integrations:connectors:source-fauna:build

airbyte-integrations/connectors/source-fauna/examples/README.md

airbyte-integrations/connectors/source-fauna/source_fauna/spec.yaml

airbyte-integrations/connectors/source-fauna/examples/config_sample.json

macmv · 2022-08-24T21:23:38Z

This latest commit should fix all the warnings produce by running

./gradlew :airbyte-integrations:connectors:source-fauna:flakeCheck

macmv · 2022-09-21T18:32:41Z

Hi @grishick, I was wondering if you could take a look at this PR sometime soon?

grishick · 2022-09-22T22:42:24Z

Looks great! I think there are several small things left to tidy up:

@marcosmarxm please make sure that the credentials for integration tests are in LastPass and are named in accordance with what the README.md says (ping me on Slack if need help with this)
@macmv in the README.md in addition to saying "create a file secrets/config.json conforming to the source_fauna/spec.yaml file" please add an example contents of config.json file or mention, which example file in examples folder to use as a base
examples/secret_config.json still has additional_columns, which, as far as I can see, have been removed from spec.yaml - need to bring this up to date as well
rebase and update the pr

grishick · 2022-09-22T23:05:57Z

@macmv could you ping me on Airbyte Slack and share config.json file that I can use to run integration tests?

* Applied suggestions to improve docs * Cleaned up the docs * Apply suggestions from code review Co-authored-by: Ewan Edwards <46354154+faunaee@users.noreply.github.com> * Update airbyte-integrations/connectors/source-fauna/source_fauna/spec.yaml Co-authored-by: Ewan Edwards <46354154+faunaee@users.noreply.github.com> Co-authored-by: Ewan Edwards <46354154+faunaee@users.noreply.github.com>

* Run ./gradlew :airbyte-integrations:connectors:source-fauna:flakeCheck * Fix all the warnings * Set additionalProperties to true to adhere to acceptance tests

* Remove custom fields from source.py * Remove custom fields from spec.yaml * Collections that support incremental sync are found correctly * Run formatter * Index values and termins are verified * Stripped additional_columns from collection config and check() * We now search for an index at the start of each sync * Add default for missing data in collection * Add a log message about the index chosen to sync an incremental stream * Add an example for a configured incremental catalog * Check test now validates the simplified check function * Remove collection name from spec.yaml and CollectionConfig * Update test_util.py to ahere to the new config * Update the first discover test to validate that we can find indexes correctly * Remove other discover tests, as they no longer apply * Full refresh test now works with simplified expanded columns * Remove unused imports * Incremental test now adheres to the find_index_for_stream system * Database test passes, so now all unit tests pass again * Remove extra fields from required section * ttl is nullable * Data defaults to an empty object * Update tests to reflect ttl and data select changes * Fix expected records. All unit tests and acceptance tests pass * Cleanup docs for find_index_for_stream * Update setup guide to reflect multiple collections

…lumns

macmv · 2022-09-23T03:15:28Z

@marcosmarxm do you also want me to send you the secrets.json file?
The examples/secret_config.json is a good example of secrets/config.json, and I've added that to the README.
Yes, that config was out of date. I've updated that as well.
Rebased.

@grishick, I just sent you a message on Slack, which contains the secrets.json file.

marcosmarxm · 2022-09-29T13:36:56Z

Thanks for the contribution @macmv

cleve-fauna · 2022-09-29T16:59:57Z

Thank you @marcosmarxm and @grishick !

* Add fauna source * Update changelog to include the correct PR * Improve docs (airbytehq#1) * Applied suggestions to improve docs (airbytehq#2) * Applied suggestions to improve docs * Cleaned up the docs * Apply suggestions from code review Co-authored-by: Ewan Edwards <46354154+faunaee@users.noreply.github.com> * Update airbyte-integrations/connectors/source-fauna/source_fauna/spec.yaml Co-authored-by: Ewan Edwards <46354154+faunaee@users.noreply.github.com> Co-authored-by: Ewan Edwards <46354154+faunaee@users.noreply.github.com> * Flake Checker (airbytehq#3) * Run ./gradlew :airbyte-integrations:connectors:source-fauna:flakeCheck * Fix all the warnings * Set additionalProperties to true to adhere to acceptance tests * Remove custom fields (airbytehq#4) * Remove custom fields from source.py * Remove custom fields from spec.yaml * Collections that support incremental sync are found correctly * Run formatter * Index values and termins are verified * Stripped additional_columns from collection config and check() * We now search for an index at the start of each sync * Add default for missing data in collection * Add a log message about the index chosen to sync an incremental stream * Add an example for a configured incremental catalog * Check test now validates the simplified check function * Remove collection name from spec.yaml and CollectionConfig * Update test_util.py to ahere to the new config * Update the first discover test to validate that we can find indexes correctly * Remove other discover tests, as they no longer apply * Full refresh test now works with simplified expanded columns * Remove unused imports * Incremental test now adheres to the find_index_for_stream system * Database test passes, so now all unit tests pass again * Remove extra fields from required section * ttl is nullable * Data defaults to an empty object * Update tests to reflect ttl and data select changes * Fix expected records. All unit tests and acceptance tests pass * Cleanup docs for find_index_for_stream * Update setup guide to reflect multiple collections * Add docs to install the fauna shell * Update examples and README to conform to the removal of additional columns Co-authored-by: Ewan Edwards <46354154+faunaee@users.noreply.github.com>

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Aug 3, 2022

octavia-squidington-iii added bounty community connectors/source/fauna labels Aug 3, 2022

sajarin added the bounty-L Maintainer program: claimable large bounty PR label Aug 5, 2022

marcosmarxm mentioned this pull request Aug 11, 2022

New Source: Fauna need sandbox account #15552

Closed

marcosmarxm requested changes Aug 16, 2022

View reviewed changes

airbyte-integrations/connectors/source-fauna/source_fauna/spec.yaml Show resolved Hide resolved

sajarin added internal and removed bounty bounty-L Maintainer program: claimable large bounty PR labels Aug 17, 2022

grishick self-assigned this Aug 18, 2022

grishick suggested changes Aug 23, 2022

View reviewed changes

airbyte-integrations/connectors/source-fauna/examples/README.md Show resolved Hide resolved

airbyte-integrations/connectors/source-fauna/source_fauna/spec.yaml Show resolved Hide resolved

grishick suggested changes Aug 23, 2022

View reviewed changes

airbyte-integrations/connectors/source-fauna/examples/config_sample.json Show resolved Hide resolved

macmv requested a review from grishick September 14, 2022 15:43

macmv and others added 8 commits September 22, 2022 20:07

Add fauna source

95d673e

Update changelog to include the correct PR

cb36f73

Improve docs (#1)

934266e

Flake Checker (#3)

cb149ea

* Run ./gradlew :airbyte-integrations:connectors:source-fauna:flakeCheck * Fix all the warnings * Set additionalProperties to true to adhere to acceptance tests

Add docs to install the fauna shell

b9cc9ff

Update examples and README to conform to the removal of additional co…

1db6115

…lumns

macmv force-pushed the source-fauna branch from ce9e904 to 1db6115 Compare September 23, 2022 03:12

grishick approved these changes Sep 26, 2022

View reviewed changes

marcosmarxm approved these changes Sep 29, 2022

View reviewed changes

marcosmarxm mentioned this pull request Sep 29, 2022

Source Fauna: add seed file information #17355

Merged

37 tasks

marcosmarxm merged commit 65e6168 into airbytehq:master Sep 29, 2022

octavia-squidington-iii mentioned this pull request Oct 5, 2022

Bump Airbyte version from 0.40.10 to 0.40.11 #17610

Merged

akashkulk mentioned this pull request Nov 16, 2022

Publish version 0.1.0 for Fauna source #19510

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 New source: Fauna #15274

🎉 New source: Fauna #15274

macmv commented Aug 3, 2022 •

edited

Loading

CLAassistant commented Aug 3, 2022 •

edited

Loading

marcosmarxm commented Aug 15, 2022

grishick left a comment

macmv commented Aug 24, 2022

macmv commented Sep 21, 2022

grishick commented Sep 22, 2022 •

edited

Loading

grishick commented Sep 22, 2022

macmv commented Sep 23, 2022 •

edited

Loading

marcosmarxm commented Sep 29, 2022

cleve-fauna commented Sep 29, 2022

🎉 New source: Fauna #15274

🎉 New source: Fauna #15274

Conversation

macmv commented Aug 3, 2022 • edited Loading

What

🚨 User Impact 🚨

Pre-merge Checklist

Tests

CLAassistant commented Aug 3, 2022 • edited Loading

marcosmarxm commented Aug 15, 2022

grishick left a comment

Choose a reason for hiding this comment

macmv commented Aug 24, 2022

macmv commented Sep 21, 2022

grishick commented Sep 22, 2022 • edited Loading

grishick commented Sep 22, 2022

macmv commented Sep 23, 2022 • edited Loading

marcosmarxm commented Sep 29, 2022

cleve-fauna commented Sep 29, 2022

macmv commented Aug 3, 2022 •

edited

Loading

CLAassistant commented Aug 3, 2022 •

edited

Loading

grishick commented Sep 22, 2022 •

edited

Loading

macmv commented Sep 23, 2022 •

edited

Loading